Note: This is test shard 6 of 8. [==========] Running 4 tests from 3 test cases. [----------] Global test environment set-up. [----------] 2 tests from AdminCliTest [ RUN ] AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig WARNING: Logging before InitGoogleLogging() is written to STDERR I1017 23:32:36.827965 18683 test_util.cc:209] Using random seed: -203035877 I1017 23:32:36.846735 18683 ts_itest-base.cc:113] Starting cluster with: I1017 23:32:36.846781 18683 ts_itest-base.cc:114] -------------- I1017 23:32:36.846801 18683 ts_itest-base.cc:115] 5 tablet servers I1017 23:32:36.846824 18683 ts_itest-base.cc:116] 3 replicas per TS I1017 23:32:36.846841 18683 ts_itest-base.cc:117] -------------- I1017 23:32:36.848917 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.18.62.254:36993 --webserver_interface=127.18.62.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:37.040443 18693 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:37.040695 18693 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:37.043011 18693 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W1017 23:32:37.043051 18693 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:37.043081 18693 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W1017 23:32:37.043107 18693 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I1017 23:32:37.051313 18693 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.18.62.254:36993 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.18.62.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:37.058400 18693 master_main.cc:80] Initializing master server... I1017 23:32:37.059983 18693 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 746673us I1017 23:32:37.061087 18693 fs_manager.cc:263] Metadata directory not provided I1017 23:32:37.061144 18693 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:37.061815 18693 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:37.061878 18693 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:37.065541 18693 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/instance: uuid: "41a36d3babc44a5b89084f8c1b4598b2" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.066057 18693 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "41a36d3babc44a5b89084f8c1b4598b2" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.074218 18693 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I1017 23:32:37.075940 18693 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:37.076328 18693 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:37.079636 18693 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I1017 23:32:37.079691 18693 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal uuid: "41a36d3babc44a5b89084f8c1b4598b2" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.079802 18693 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:37.112352 18693 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:37.113854 18693 master_main.cc:83] Starting Master server... I1017 23:32:37.153837 18693 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.254:36993 I1017 23:32:37.154304 18693 webserver.cc:175] Starting webserver on 127.18.62.254:0 I1017 23:32:37.154350 18693 webserver.cc:186] Document root disabled I1017 23:32:37.155603 18693 webserver.cc:313] Webserver started. Bound to: http://127.18.62.254:52888/ I1017 23:32:37.156236 18693 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb I1017 23:32:37.159775 18752 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:37.160863 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 18693 I1017 23:32:37.162034 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.18.62.193:0 --local_ip_for_outbound_sockets=127.18.62.193 --webserver_interface=127.18.62.193 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:37.173425 18752 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Bootstrap starting. I1017 23:32:37.175762 18752 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: No blocks or log segments found. Creating new log. I1017 23:32:37.176555 18752 log.cc:525] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Log is configured to *not* fsync() on all Append() calls I1017 23:32:37.179219 18752 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: No bootstrap required, opened a new log I1017 23:32:37.185665 18752 raft_consensus.cc:339] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:37.185900 18752 raft_consensus.cc:365] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:37.185976 18752 raft_consensus.cc:704] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a36d3babc44a5b89084f8c1b4598b2, State: Initialized, Role: FOLLOWER I1017 23:32:37.186561 18752 consensus_queue.cc:229] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:37.186717 18752 raft_consensus.cc:379] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1017 23:32:37.186799 18752 raft_consensus.cc:471] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1017 23:32:37.186880 18752 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Snoozing failure detection for 2.122s (starting election) I1017 23:32:37.186972 18752 raft_consensus.cc:2892] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:37.188586 18752 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:37.189018 18752 leader_election.cc:278] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1017 23:32:37.190735 18755 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Snoozing failure detection for 1.677s (election complete) I1017 23:32:37.191218 18755 raft_consensus.cc:2646] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Leader election won for term 1 I1017 23:32:37.192683 18755 raft_consensus.cc:666] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 LEADER]: Becoming Leader. State: Replica: 41a36d3babc44a5b89084f8c1b4598b2, State: Running, Role: LEADER I1017 23:32:37.193126 18755 consensus_queue.cc:206] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:37.195942 18752 sys_catalog.cc:337] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "41a36d3babc44a5b89084f8c1b4598b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } } I1017 23:32:37.196771 18752 sys_catalog.cc:340] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: This master's current role is: LEADER I1017 23:32:37.196772 18757 sys_catalog.cc:337] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a36d3babc44a5b89084f8c1b4598b2. Latest consensus state: current_term: 1 leader_uuid: "41a36d3babc44a5b89084f8c1b4598b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } } I1017 23:32:37.196774 18756 sys_catalog.cc:337] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a36d3babc44a5b89084f8c1b4598b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } } I1017 23:32:37.197490 18757 sys_catalog.cc:340] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: This master's current role is: LEADER I1017 23:32:37.197589 18756 sys_catalog.cc:340] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: This master's current role is: LEADER I1017 23:32:37.200361 18752 sys_catalog.cc:424] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: configured and running, proceeding with master startup. I1017 23:32:37.201465 18759 catalog_manager.cc:1026] Loading table and tablet metadata into memory... I1017 23:32:37.201869 18693 master_main.cc:86] Master server successfully started. I1017 23:32:37.207659 18759 catalog_manager.cc:1037] Initializing Kudu internal certificate authority... I1017 23:32:37.241488 18759 catalog_manager.cc:924] Generated new certificate authority record I1017 23:32:37.243059 18759 catalog_manager.cc:1046] Loading token signing keys... I1017 23:32:37.269748 18759 catalog_manager.cc:4062] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:37.331192 18753 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:37.331409 18753 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:37.333387 18753 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:37.333499 18753 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.193 I1017 23:32:37.340907 18753 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.18.62.193 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:37.349169 18753 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:37.350328 18753 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 746673us I1017 23:32:37.351348 18753 fs_manager.cc:263] Metadata directory not provided I1017 23:32:37.351414 18753 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:37.352107 18753 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:37.352172 18753 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:37.355429 18753 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "92a7d6efbf034107a3f7e47dce492bb8" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.355926 18753 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "92a7d6efbf034107a3f7e47dce492bb8" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.364800 18753 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s I1017 23:32:37.366094 18753 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:37.366525 18753 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:37.370594 18753 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I1017 23:32:37.370661 18753 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal uuid: "92a7d6efbf034107a3f7e47dce492bb8" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.370782 18753 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:37.426836 18753 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:37.429049 18753 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:37.429549 18753 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:37.533969 18753 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.193:53195 I1017 23:32:37.534509 18753 webserver.cc:175] Starting webserver on 127.18.62.193:0 I1017 23:32:37.534548 18753 webserver.cc:186] Document root disabled I1017 23:32:37.535962 18753 webserver.cc:313] Webserver started. Bound to: http://127.18.62.193:43508/ I1017 23:32:37.536622 18753 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/info.pb I1017 23:32:37.539463 18753 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:37.545311 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18753 I1017 23:32:37.546262 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.18.62.194:0 --local_ip_for_outbound_sockets=127.18.62.194 --webserver_interface=127.18.62.194 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:37.556085 18882 heartbeater.cc:346] Connected to a master server at 127.18.62.254:36993 I1017 23:32:37.556246 18882 heartbeater.cc:426] Registering TS with master... I1017 23:32:37.556866 18882 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:37.558630 18719 ts_manager.cc:98] Registered new tserver with Master: 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193:53195) I1017 23:32:37.560796 18719 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.193:53950 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:37.722426 18886 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:37.722673 18886 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:37.724681 18886 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:37.724776 18886 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.194 I1017 23:32:37.732132 18886 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.18.62.194 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:37.740353 18886 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:37.741592 18886 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 746673us I1017 23:32:37.742597 18886 fs_manager.cc:263] Metadata directory not provided I1017 23:32:37.742653 18886 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:37.743286 18886 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:37.743351 18886 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:37.746515 18886 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "2bd18a72ed7d447da0448d9193fa13af" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.747021 18886 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "2bd18a72ed7d447da0448d9193fa13af" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.754963 18886 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I1017 23:32:37.756283 18886 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:37.756705 18886 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:37.760879 18886 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I1017 23:32:37.760933 18886 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal uuid: "2bd18a72ed7d447da0448d9193fa13af" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:37.761046 18886 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:37.856005 18886 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:37.857944 18886 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:37.858314 18886 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:37.948843 18886 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.194:58520 I1017 23:32:37.949280 18886 webserver.cc:175] Starting webserver on 127.18.62.194:0 I1017 23:32:37.949311 18886 webserver.cc:186] Document root disabled I1017 23:32:37.950031 18886 webserver.cc:313] Webserver started. Bound to: http://127.18.62.194:36742/ I1017 23:32:37.950615 18886 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/info.pb I1017 23:32:37.951226 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18886 I1017 23:32:37.952113 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.18.62.195:0 --local_ip_for_outbound_sockets=127.18.62.195 --webserver_interface=127.18.62.195 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:37.953140 18886 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:37.966065 19005 heartbeater.cc:346] Connected to a master server at 127.18.62.254:36993 I1017 23:32:37.966213 19005 heartbeater.cc:426] Registering TS with master... I1017 23:32:37.967057 19005 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:37.968273 18719 ts_manager.cc:98] Registered new tserver with Master: 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194:58520) I1017 23:32:37.969604 18719 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.194:39293 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:38.122051 19007 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:38.122262 19007 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:38.124266 19007 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:38.124361 19007 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.195 I1017 23:32:38.131729 19007 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.18.62.195 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:38.139902 19007 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:38.141082 19007 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 747173us I1017 23:32:38.142086 19007 fs_manager.cc:263] Metadata directory not provided I1017 23:32:38.142141 19007 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:38.142812 19007 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:38.142874 19007 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:38.146024 19007 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "a0f0d49ea68149b9aef79c2d126bcc77" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.146567 19007 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "a0f0d49ea68149b9aef79c2d126bcc77" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.154530 19007 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I1017 23:32:38.155829 19007 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:38.156219 19007 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:38.159876 19007 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I1017 23:32:38.159938 19007 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal uuid: "a0f0d49ea68149b9aef79c2d126bcc77" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.160058 19007 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:38.187165 19007 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:38.188637 19007 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:38.189008 19007 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:38.278203 19007 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.195:46826 I1017 23:32:38.278658 19007 webserver.cc:175] Starting webserver on 127.18.62.195:0 I1017 23:32:38.278695 19007 webserver.cc:186] Document root disabled I1017 23:32:38.279940 19007 webserver.cc:313] Webserver started. Bound to: http://127.18.62.195:37363/ I1017 23:32:38.280544 19007 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/info.pb I1017 23:32:38.283090 19007 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:38.283700 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19007 I1017 23:32:38.284663 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.18.62.196:0 --local_ip_for_outbound_sockets=127.18.62.196 --webserver_interface=127.18.62.196 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:38.296203 19127 heartbeater.cc:346] Connected to a master server at 127.18.62.254:36993 I1017 23:32:38.296375 19127 heartbeater.cc:426] Registering TS with master... I1017 23:32:38.296977 19127 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:38.298156 18719 ts_manager.cc:98] Registered new tserver with Master: a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195:46826) I1017 23:32:38.299535 18719 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.195:38735 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:38.449163 19130 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:38.449394 19130 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:38.451405 19130 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:38.451542 19130 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.196 I1017 23:32:38.458938 19130 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.18.62.196 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:38.466980 19130 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:38.468097 19130 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 747173us I1017 23:32:38.469128 19130 fs_manager.cc:263] Metadata directory not provided I1017 23:32:38.469193 19130 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:38.469882 19130 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:38.469947 19130 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:38.473080 19130 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "57c44921a8064f9c84f3646d8158b003" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.473632 19130 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "57c44921a8064f9c84f3646d8158b003" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.481426 19130 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.002s sys 0.003s I1017 23:32:38.482709 19130 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:38.483105 19130 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:38.487234 19130 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I1017 23:32:38.487296 19130 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/wal uuid: "57c44921a8064f9c84f3646d8158b003" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.487416 19130 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:38.517202 19130 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:38.518641 19130 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:38.518967 19130 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:38.563454 18882 heartbeater.cc:467] Master 127.18.62.254:36993 was elected leader, sending a full tablet report... I1017 23:32:38.609724 19130 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.196:49688 I1017 23:32:38.610158 19130 webserver.cc:175] Starting webserver on 127.18.62.196:0 I1017 23:32:38.610194 19130 webserver.cc:186] Document root disabled I1017 23:32:38.611459 19130 webserver.cc:313] Webserver started. Bound to: http://127.18.62.196:57454/ I1017 23:32:38.612020 19130 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-3/data/info.pb I1017 23:32:38.614097 19130 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:38.615788 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19130 I1017 23:32:38.616701 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.18.62.197:0 --local_ip_for_outbound_sockets=127.18.62.197 --webserver_interface=127.18.62.197 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:38.627229 19249 heartbeater.cc:346] Connected to a master server at 127.18.62.254:36993 I1017 23:32:38.627378 19249 heartbeater.cc:426] Registering TS with master... I1017 23:32:38.627924 19249 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:38.629012 18719 ts_manager.cc:98] Registered new tserver with Master: 57c44921a8064f9c84f3646d8158b003 (127.18.62.196:49688) I1017 23:32:38.629902 18719 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.196:34786 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:38.783452 19252 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:38.783697 19252 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:38.785735 19252 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:38.785830 19252 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.197 I1017 23:32:38.793282 19252 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.18.62.197 --webserver_port=0 --tserver_master_addrs=127.18.62.254:36993 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:38.801162 19252 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:38.802311 19252 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 747173us I1017 23:32:38.803303 19252 fs_manager.cc:263] Metadata directory not provided I1017 23:32:38.803356 19252 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:38.804025 19252 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:38.804085 19252 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:38.807227 19252 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "b8818ab3ec794a36a38bce58c4599155" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.807780 19252 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "b8818ab3ec794a36a38bce58c4599155" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.815703 19252 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s I1017 23:32:38.817011 19252 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:38.817401 19252 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:38.821571 19252 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I1017 23:32:38.821625 19252 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/wal uuid: "b8818ab3ec794a36a38bce58c4599155" format_stamp: "Formatted at 2018-10-18 06:32:38 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:38.821744 19252 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:38.849867 19252 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:38.851820 19252 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:38.852180 19252 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:38.942952 19252 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.197:53959 I1017 23:32:38.943397 19252 webserver.cc:175] Starting webserver on 127.18.62.197:0 I1017 23:32:38.943434 19252 webserver.cc:186] Document root disabled I1017 23:32:38.944164 19252 webserver.cc:313] Webserver started. Bound to: http://127.18.62.197:55680/ I1017 23:32:38.944759 19252 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/ts-4/data/info.pb I1017 23:32:38.948621 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19252 I1017 23:32:38.949759 19252 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:38.960629 19371 heartbeater.cc:346] Connected to a master server at 127.18.62.254:36993 I1017 23:32:38.960778 19371 heartbeater.cc:426] Registering TS with master... I1017 23:32:38.961637 19371 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:38.962713 18719 ts_manager.cc:98] Registered new tserver with Master: b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959) I1017 23:32:38.963634 18719 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.197:46694 I1017 23:32:38.969780 18683 external_mini_cluster.cc:535] 5 TS(s) registered with all masters I1017 23:32:38.971717 19005 heartbeater.cc:467] Master 127.18.62.254:36993 was elected leader, sending a full tablet report... I1017 23:32:38.988833 18719 catalog_manager.cc:1366] Servicing CreateTable request from {username='jenkins'} at 127.18.62.254:48802: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I1017 23:32:39.016564 18819 tablet_service.cc:765] Processing CreateTablet for tablet 1063e4b7405f470f859a82a5f2e0098a (table=TestTable [id=149a6d5622424e4aae2275a869d6d3a8]), partition=RANGE (key) PARTITION UNBOUNDED I1017 23:32:39.017426 18819 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1063e4b7405f470f859a82a5f2e0098a. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:39.018024 18942 tablet_service.cc:765] Processing CreateTablet for tablet 1063e4b7405f470f859a82a5f2e0098a (table=TestTable [id=149a6d5622424e4aae2275a869d6d3a8]), partition=RANGE (key) PARTITION UNBOUNDED I1017 23:32:39.018640 18942 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1063e4b7405f470f859a82a5f2e0098a. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:39.018882 19308 tablet_service.cc:765] Processing CreateTablet for tablet 1063e4b7405f470f859a82a5f2e0098a (table=TestTable [id=149a6d5622424e4aae2275a869d6d3a8]), partition=RANGE (key) PARTITION UNBOUNDED I1017 23:32:39.019744 19308 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1063e4b7405f470f859a82a5f2e0098a. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:39.029590 18942 ts_tablet_manager.cc:1173] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Registered tablet (data state: TABLET_DATA_READY) I1017 23:32:39.030418 19391 ts_tablet_manager.cc:1028] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Bootstrapping tablet I1017 23:32:39.031216 19391 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Bootstrap starting. I1017 23:32:39.032037 18819 ts_tablet_manager.cc:1173] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Registered tablet (data state: TABLET_DATA_READY) I1017 23:32:39.032869 19392 ts_tablet_manager.cc:1028] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Bootstrapping tablet I1017 23:32:39.033224 19392 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Bootstrap starting. I1017 23:32:39.033962 19308 ts_tablet_manager.cc:1173] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Registered tablet (data state: TABLET_DATA_READY) I1017 23:32:39.034320 19391 tablet_bootstrap.cc:592] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: No blocks or log segments found. Creating new log. I1017 23:32:39.035414 19393 ts_tablet_manager.cc:1028] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Bootstrapping tablet I1017 23:32:39.035784 19393 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Bootstrap starting. I1017 23:32:39.036269 19391 log.cc:525] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Log is configured to *not* fsync() on all Append() calls I1017 23:32:39.036593 19392 tablet_bootstrap.cc:592] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: No blocks or log segments found. Creating new log. I1017 23:32:39.037317 19392 log.cc:525] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Log is configured to *not* fsync() on all Append() calls I1017 23:32:39.038558 19391 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: No bootstrap required, opened a new log I1017 23:32:39.038563 19393 tablet_bootstrap.cc:592] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: No blocks or log segments found. Creating new log. I1017 23:32:39.038749 19391 ts_tablet_manager.cc:1045] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s I1017 23:32:39.039176 19393 log.cc:525] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Log is configured to *not* fsync() on all Append() calls I1017 23:32:39.044054 19393 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: No bootstrap required, opened a new log I1017 23:32:39.044224 19393 ts_tablet_manager.cc:1045] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s I1017 23:32:39.044769 19392 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: No bootstrap required, opened a new log I1017 23:32:39.044973 19392 ts_tablet_manager.cc:1045] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s I1017 23:32:39.047863 19391 raft_consensus.cc:339] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.048388 19391 raft_consensus.cc:365] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:39.048493 19391 raft_consensus.cc:704] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd18a72ed7d447da0448d9193fa13af, State: Initialized, Role: FOLLOWER I1017 23:32:39.049373 19391 consensus_queue.cc:229] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.050750 19393 raft_consensus.cc:339] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.051102 19393 raft_consensus.cc:365] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:39.051178 19393 raft_consensus.cc:704] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8818ab3ec794a36a38bce58c4599155, State: Initialized, Role: FOLLOWER I1017 23:32:39.051537 19391 ts_tablet_manager.cc:1073] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s I1017 23:32:39.051581 19393 consensus_queue.cc:229] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.053668 19371 heartbeater.cc:467] Master 127.18.62.254:36993 was elected leader, sending a full tablet report... I1017 23:32:39.054729 19393 ts_tablet_manager.cc:1073] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s I1017 23:32:39.056243 19392 raft_consensus.cc:339] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.056541 19392 raft_consensus.cc:365] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:39.056610 19392 raft_consensus.cc:704] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a7d6efbf034107a3f7e47dce492bb8, State: Initialized, Role: FOLLOWER I1017 23:32:39.056979 19392 consensus_queue.cc:229] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.059295 19392 ts_tablet_manager.cc:1073] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s I1017 23:32:39.059614 19399 raft_consensus.cc:471] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:39.059717 19399 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 1.524s (starting election) I1017 23:32:39.059818 19399 raft_consensus.cc:493] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.061064 19399 leader_election.cc:251] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959) I1017 23:32:39.061475 19399 leader_election.cc:251] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194:58520) I1017 23:32:39.069945 19328 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "1063e4b7405f470f859a82a5f2e0098a" candidate_uuid: "92a7d6efbf034107a3f7e47dce492bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8818ab3ec794a36a38bce58c4599155" is_pre_election: true I1017 23:32:39.070675 19328 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Snoozing failure detection for 2.086s (vote granted) I1017 23:32:39.070775 19328 raft_consensus.cc:2352] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a7d6efbf034107a3f7e47dce492bb8 in term 0. I1017 23:32:39.070361 18962 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "1063e4b7405f470f859a82a5f2e0098a" candidate_uuid: "92a7d6efbf034107a3f7e47dce492bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd18a72ed7d447da0448d9193fa13af" is_pre_election: true I1017 23:32:39.071106 18962 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Snoozing failure detection for 2.111s (vote granted) I1017 23:32:39.071194 18962 raft_consensus.cc:2352] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a7d6efbf034107a3f7e47dce492bb8 in term 0. I1017 23:32:39.071458 18770 leader_election.cc:387] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959) I1017 23:32:39.071568 18770 leader_election.cc:278] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I1017 23:32:39.071933 18773 leader_election.cc:387] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194:58520) I1017 23:32:39.071936 19399 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 1.881s (election complete) I1017 23:32:39.072054 19399 raft_consensus.cc:2646] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1017 23:32:39.072156 19399 raft_consensus.cc:471] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:39.072217 19399 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 1.613s (starting election) I1017 23:32:39.072276 19399 raft_consensus.cc:2892] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:39.073909 19399 raft_consensus.cc:493] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.074242 19399 leader_election.cc:251] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 election: Requesting vote from peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959) I1017 23:32:39.074394 19399 leader_election.cc:251] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 election: Requesting vote from peer 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194:58520) I1017 23:32:39.074867 19328 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "1063e4b7405f470f859a82a5f2e0098a" candidate_uuid: "92a7d6efbf034107a3f7e47dce492bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8818ab3ec794a36a38bce58c4599155" I1017 23:32:39.075002 19328 raft_consensus.cc:2892] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:39.075109 19328 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Snoozing failure detection for 1.583s (vote granted) I1017 23:32:39.075284 18962 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "1063e4b7405f470f859a82a5f2e0098a" candidate_uuid: "92a7d6efbf034107a3f7e47dce492bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bd18a72ed7d447da0448d9193fa13af" I1017 23:32:39.075389 18962 raft_consensus.cc:2892] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:39.075470 18962 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Snoozing failure detection for 2.199s (vote granted) I1017 23:32:39.076580 19328 raft_consensus.cc:2352] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a7d6efbf034107a3f7e47dce492bb8 in term 1. I1017 23:32:39.076953 18962 raft_consensus.cc:2352] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a7d6efbf034107a3f7e47dce492bb8 in term 1. I1017 23:32:39.077011 18770 leader_election.cc:387] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 election: Vote granted by peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959) I1017 23:32:39.077073 18770 leader_election.cc:278] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1017 23:32:39.077275 19399 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 2.199s (election complete) I1017 23:32:39.077389 19399 raft_consensus.cc:2646] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 FOLLOWER]: Leader election won for term 1 I1017 23:32:39.077463 18773 leader_election.cc:387] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 1 election: Vote granted by peer 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194:58520) I1017 23:32:39.077854 19399 raft_consensus.cc:666] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 92a7d6efbf034107a3f7e47dce492bb8, State: Running, Role: LEADER I1017 23:32:39.078384 19399 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } I1017 23:32:39.083022 18718 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: term changed from 0 to 1, leader changed from to 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193). New cstate: current_term: 1 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } health_report { overall_health: UNKNOWN } } } I1017 23:32:39.104934 18683 external_mini_cluster.cc:535] 5 TS(s) registered with all masters I1017 23:32:39.106304 18683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 92a7d6efbf034107a3f7e47dce492bb8 to finish bootstrapping I1017 23:32:39.111414 18683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2bd18a72ed7d447da0448d9193fa13af to finish bootstrapping I1017 23:32:39.116971 18683 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a0f0d49ea68149b9aef79c2d126bcc77 to finish bootstrapping I1017 23:32:39.124038 18683 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 57c44921a8064f9c84f3646d8158b003 to finish bootstrapping I1017 23:32:39.130142 18683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b8818ab3ec794a36a38bce58c4599155 to finish bootstrapping I1017 23:32:39.135136 18683 kudu-admin-test.cc:797] Waiting for Master to see the current replicas... I1017 23:32:39.136449 18683 kudu-admin-test.cc:804] Tablet locations: tablet_id: "1063e4b7405f470f859a82a5f2e0098a" replicas { ts_info { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" rpc_addresses { host: "127.18.62.197" port: 53959 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" rpc_addresses { host: "127.18.62.193" port: 53195 } } role: LEADER } replicas { ts_info { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" rpc_addresses { host: "127.18.62.194" port: 58520 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W1017 23:32:39.196887 19372 tablet.cc:1744] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155: Can't schedule compaction. Clean time has not been advanced past its initial value. W1017 23:32:39.204082 19006 tablet.cc:1744] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af: Can't schedule compaction. Clean time has not been advanced past its initial value. W1017 23:32:39.290555 18883 tablet.cc:1744] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Can't schedule compaction. Clean time has not been advanced past its initial value. I1017 23:32:39.301479 19127 heartbeater.cc:467] Master 127.18.62.254:36993 was elected leader, sending a full tablet report... I1017 23:32:39.622692 18962 raft_consensus.cc:1180] T 1063e4b7405f470f859a82a5f2e0098a P 2bd18a72ed7d447da0448d9193fa13af [term 1 FOLLOWER]: Refusing update from remote peer 92a7d6efbf034107a3f7e47dce492bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1017 23:32:39.623615 19407 consensus_queue.cc:976] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1017 23:32:39.630295 19328 raft_consensus.cc:1180] T 1063e4b7405f470f859a82a5f2e0098a P b8818ab3ec794a36a38bce58c4599155 [term 1 FOLLOWER]: Refusing update from remote peer 92a7d6efbf034107a3f7e47dce492bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I1017 23:32:39.631820 19249 heartbeater.cc:467] Master 127.18.62.254:36993 was elected leader, sending a full tablet report... I1017 23:32:39.631820 19407 consensus_queue.cc:976] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1017 23:32:39.634167 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19252 I1017 23:32:39.635006 19368 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:39.681087 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19252 W1017 23:32:39.689774 18770 connection.cc:512] client connection to 127.18.62.197:53959 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.197:53959): Connection reset by peer (error 104) W1017 23:32:39.689963 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: failed to read from TLS socket (remote: 127.18.62.197:53959): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W1017 23:32:39.690212 18701 connection.cc:512] server connection from 127.18.62.197:46694 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.197:46694): Connection reset by peer (error 104) W1017 23:32:39.690321 18701 connection.cc:148] Shutting down server connection from 127.18.62.197:46694 with pending inbound data (52/387 bytes received, last active 0 ns ago, status=Network error: failed to read from TLS socket (remote: 127.18.62.197:46694): Connection reset by peer (error 104)) I1017 23:32:39.690843 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18886 I1017 23:32:39.691635 19002 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:39.735854 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18886 W1017 23:32:39.744832 18701 connection.cc:512] server connection from 127.18.62.194:39293 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.194:39293): Connection reset by peer (error 104) W1017 23:32:39.744983 18773 connection.cc:512] client connection to 127.18.62.194:58520 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.194:58520): Connection reset by peer (error 104) I1017 23:32:39.746377 18839 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194:58520): Closing peer: 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:39.746444 18839 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Closing peer: b8818ab3ec794a36a38bce58c4599155 I1017 23:32:39.746783 18839 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } W1017 23:32:39.748491 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: Client connection negotiation failed: client connection to 127.18.62.197:53959: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W1017 23:32:40.126020 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: Client connection negotiation failed: client connection to 127.18.62.197:53959: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W1017 23:32:40.615670 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: Client connection negotiation failed: client connection to 127.18.62.197:53959: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W1017 23:32:41.013872 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: Client connection negotiation failed: client connection to 127.18.62.197:53959: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W1017 23:32:41.577878 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: Client connection negotiation failed: client connection to 127.18.62.197:53959: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I1017 23:32:41.746779 18683 kudu-admin-test.cc:826] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W1017 23:32:41.970350 19426 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:42.003317 18770 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Couldn't send request to peer b8818ab3ec794a36a38bce58c4599155 for tablet 1063e4b7405f470f859a82a5f2e0098a. Status: Network error: Client connection negotiation failed: client connection to 127.18.62.197:53959: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. I1017 23:32:42.022430 18839 tablet_service.cc:1083] Received UnsafeChangeConfig RPC: dest_uuid: "92a7d6efbf034107a3f7e47dce492bb8" tablet_id: "1063e4b7405f470f859a82a5f2e0098a" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" } } from {username='jenkins'} at 127.18.62.193:58170 W1017 23:32:42.022682 18839 raft_consensus.cc:2027] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 LEADER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } W1017 23:32:42.022982 18839 raft_consensus.cc:2126] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true I1017 23:32:42.023322 18839 raft_consensus.cc:2887] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 LEADER]: Stepping down as leader of term 1 I1017 23:32:42.023375 18839 raft_consensus.cc:704] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 92a7d6efbf034107a3f7e47dce492bb8, State: Running, Role: LEADER I1017 23:32:42.023600 18839 consensus_queue.cc:229] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } I1017 23:32:42.023735 18839 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer b8818ab3ec794a36a38bce58c4599155 (127.18.62.197:53959): Closing peer: b8818ab3ec794a36a38bce58c4599155 I1017 23:32:42.023854 18839 raft_consensus.cc:2892] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 1 FOLLOWER]: Advancing to term 2 I1017 23:32:42.026106 18839 raft_consensus.cc:2962] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true I1017 23:32:42.071890 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 18693 I1017 23:32:42.073004 18749 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:42.139706 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 18693 W1017 23:32:42.153718 18773 connection.cc:512] client connection to 127.18.62.254:36993 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.254:36993): Connection reset by peer (error 104) W1017 23:32:42.153766 19018 connection.cc:512] client connection to 127.18.62.254:36993 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.254:36993): Connection reset by peer (error 104) W1017 23:32:42.153767 19140 connection.cc:512] client connection to 127.18.62.254:36993 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.254:36993): Connection reset by peer (error 104) I1017 23:32:42.154251 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.18.62.254:36993 --webserver_interface=127.18.62.254 --webserver_port=52888 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W1017 23:32:42.307580 19127 heartbeater.cc:587] Failed to heartbeat to 127.18.62.254:36993: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.62.254:36993: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:42.347293 19445 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:42.347553 19445 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:42.349865 19445 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W1017 23:32:42.349915 19445 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:42.349964 19445 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W1017 23:32:42.350014 19445 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I1017 23:32:42.358353 19445 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.18.62.254:36993 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.18.62.254 --webserver_port=52888 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:42.365535 19445 master_main.cc:80] Initializing master server... I1017 23:32:42.367108 19445 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 749173us I1017 23:32:42.368224 19445 fs_manager.cc:263] Metadata directory not provided I1017 23:32:42.368300 19445 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:42.375315 19445 fs_manager.cc:399] Time spent opening directory manager: real 0.005s user 0.001s sys 0.002s I1017 23:32:42.376266 19445 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:42.376708 19445 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:42.380408 19445 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I1017 23:32:42.380486 19445 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal uuid: "41a36d3babc44a5b89084f8c1b4598b2" format_stamp: "Formatted at 2018-10-18 06:32:37 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:42.381846 19445 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:42.417167 19445 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:42.418203 19445 master_main.cc:83] Starting Master server... I1017 23:32:42.457178 19445 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.254:36993 I1017 23:32:42.457684 19445 webserver.cc:175] Starting webserver on 127.18.62.254:52888 I1017 23:32:42.457726 19445 webserver.cc:186] Document root disabled I1017 23:32:42.458981 19445 webserver.cc:313] Webserver started. Bound to: http://127.18.62.254:52888/ I1017 23:32:42.459746 19445 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb I1017 23:32:42.466431 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 19445 I1017 23:32:42.468601 19505 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Bootstrap starting. I1017 23:32:42.482290 19505 log.cc:525] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Log is configured to *not* fsync() on all Append() calls I1017 23:32:42.491619 19505 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I1017 23:32:42.492112 19505 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Bootstrap complete. I1017 23:32:42.501561 19505 raft_consensus.cc:339] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:42.502251 19505 raft_consensus.cc:704] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a36d3babc44a5b89084f8c1b4598b2, State: Initialized, Role: FOLLOWER I1017 23:32:42.502732 19505 consensus_queue.cc:229] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:42.502884 19505 raft_consensus.cc:379] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1017 23:32:42.502967 19505 raft_consensus.cc:471] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1017 23:32:42.503051 19505 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Snoozing failure detection for 1.675s (starting election) I1017 23:32:42.503145 19505 raft_consensus.cc:2892] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 1 FOLLOWER]: Advancing to term 2 I1017 23:32:42.506755 19505 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:42.507128 19505 leader_election.cc:278] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I1017 23:32:42.508364 19509 raft_consensus.cc:2840] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Snoozing failure detection for 2.215s (election complete) I1017 23:32:42.508649 19509 raft_consensus.cc:2646] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 2 FOLLOWER]: Leader election won for term 2 I1017 23:32:42.510437 19509 raft_consensus.cc:666] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [term 2 LEADER]: Becoming Leader. State: Replica: 41a36d3babc44a5b89084f8c1b4598b2, State: Running, Role: LEADER I1017 23:32:42.510841 19509 consensus_queue.cc:206] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } I1017 23:32:42.512619 19505 sys_catalog.cc:337] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "41a36d3babc44a5b89084f8c1b4598b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } } I1017 23:32:42.512962 19505 sys_catalog.cc:340] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: This master's current role is: LEADER I1017 23:32:42.512624 19510 sys_catalog.cc:337] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41a36d3babc44a5b89084f8c1b4598b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } } I1017 23:32:42.512619 19511 sys_catalog.cc:337] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a36d3babc44a5b89084f8c1b4598b2. Latest consensus state: current_term: 2 leader_uuid: "41a36d3babc44a5b89084f8c1b4598b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a36d3babc44a5b89084f8c1b4598b2" member_type: VOTER } } I1017 23:32:42.514258 19510 sys_catalog.cc:340] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: This master's current role is: LEADER I1017 23:32:42.514587 19511 sys_catalog.cc:340] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: This master's current role is: LEADER I1017 23:32:42.516685 19505 sys_catalog.cc:424] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2 [sys.catalog]: configured and running, proceeding with master startup. I1017 23:32:42.517995 19513 catalog_manager.cc:1026] Loading table and tablet metadata into memory... I1017 23:32:42.518216 19445 master_main.cc:86] Master server successfully started. I1017 23:32:42.521981 19513 catalog_manager.cc:343] Loaded metadata for table TestTable [id=149a6d5622424e4aae2275a869d6d3a8] I1017 23:32:42.524396 19513 catalog_manager.cc:397] Loaded metadata for tablet 1063e4b7405f470f859a82a5f2e0098a (table TestTable [id=149a6d5622424e4aae2275a869d6d3a8]) I1017 23:32:42.524606 19513 catalog_manager.cc:1037] Initializing Kudu internal certificate authority... I1017 23:32:42.528334 19513 catalog_manager.cc:1046] Loading token signing keys... I1017 23:32:42.529588 19513 catalog_manager.cc:4073] T 00000000000000000000000000000000 P 41a36d3babc44a5b89084f8c1b4598b2: Loaded TSK: 0 I1017 23:32:42.648320 19472 master_service.cc:209] Got heartbeat from unknown tserver (permanent_uuid: "57c44921a8064f9c84f3646d8158b003" instance_seqno: 1539844358591384) as {username='jenkins'} at 127.18.62.196:51237; Asking this server to re-register. I1017 23:32:42.649581 19249 heartbeater.cc:426] Registering TS with master... I1017 23:32:42.649718 19249 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:42.650754 19472 ts_manager.cc:98] Registered new tserver with Master: 57c44921a8064f9c84f3646d8158b003 (127.18.62.196:49688) I1017 23:32:43.033463 19472 master_service.cc:209] Got heartbeat from unknown tserver (permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" instance_seqno: 1539844357513048) as {username='jenkins'} at 127.18.62.193:44893; Asking this server to re-register. I1017 23:32:43.034050 18882 heartbeater.cc:426] Registering TS with master... I1017 23:32:43.034209 18882 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:43.035107 19472 ts_manager.cc:98] Registered new tserver with Master: 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193:53195) I1017 23:32:43.312355 19127 heartbeater.cc:346] Connected to a master server at 127.18.62.254:36993 I1017 23:32:43.313063 19472 master_service.cc:209] Got heartbeat from unknown tserver (permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" instance_seqno: 1539844358260509) as {username='jenkins'} at 127.18.62.195:37118; Asking this server to re-register. I1017 23:32:43.313577 19127 heartbeater.cc:426] Registering TS with master... I1017 23:32:43.313680 19127 heartbeater.cc:475] Master 127.18.62.254:36993 requested a full tablet report, sending... I1017 23:32:43.314347 19472 ts_manager.cc:98] Registered new tserver with Master: a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195:46826) I1017 23:32:43.527259 19522 raft_consensus.cc:471] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I1017 23:32:43.527323 19522 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 1.746s (starting election) I1017 23:32:43.527391 19522 raft_consensus.cc:493] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true I1017 23:32:43.527587 19522 leader_election.cc:278] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I1017 23:32:43.527978 19522 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 2.093s (election complete) I1017 23:32:43.528033 19522 raft_consensus.cc:2646] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 2 FOLLOWER]: Leader pre-election won for term 3 I1017 23:32:43.528077 19522 raft_consensus.cc:471] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I1017 23:32:43.528115 19522 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 1.922s (starting election) I1017 23:32:43.528158 19522 raft_consensus.cc:2892] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 2 FOLLOWER]: Advancing to term 3 I1017 23:32:43.529618 19522 raft_consensus.cc:493] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true I1017 23:32:43.529750 19522 leader_election.cc:278] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I1017 23:32:43.529920 19523 raft_consensus.cc:2840] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Snoozing failure detection for 2.206s (election complete) I1017 23:32:43.529992 19523 raft_consensus.cc:2646] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 FOLLOWER]: Leader election won for term 3 I1017 23:32:43.530138 19523 raft_consensus.cc:666] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Becoming Leader. State: Replica: 92a7d6efbf034107a3f7e47dce492bb8, State: Running, Role: LEADER I1017 23:32:43.530400 19523 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true I1017 23:32:43.531662 19523 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } } I1017 23:32:43.532006 19472 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193). New cstate: current_term: 3 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8818ab3ec794a36a38bce58c4599155" member_type: VOTER last_known_addr { host: "127.18.62.197" port: 53959 } } peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bd18a72ed7d447da0448d9193fa13af" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 58520 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true } W1017 23:32:43.534456 19523 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.ChangeConfig from 127.18.62.193:58167 (request call id 48) took 3788 ms (3.79 s). Client timeout 2000 ms (2 s) W1017 23:32:43.535054 19523 rpcz_store.cc:259] Trace: 1017 23:32:39.745556 (+ 0us) service_pool.cc:162] Inserting onto call queue 1017 23:32:39.745610 (+ 54us) service_pool.cc:221] Handling call 1017 23:32:39.747342 (+ 1732us) log.cc:587] Serialized 379 byte log entry 1017 23:32:43.534374 (+3787032us) inbound_call.cc:162] Queueing success response Metrics: {"dns_us":74,"raft.queue_time_us":67,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":163} I1017 23:32:43.535261 19523 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194) evicted, VOTER b8818ab3ec794a36a38bce58c4599155 (127.18.62.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } unsafe_config_change: true } I1017 23:32:43.541636 19472 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: config changed from index -1 to 3, VOTER 2bd18a72ed7d447da0448d9193fa13af (127.18.62.194) evicted, VOTER b8818ab3ec794a36a38bce58c4599155 (127.18.62.197) evicted, no longer has a pending config: VOTER 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193). New cstate: current_term: 3 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I1017 23:32:43.543670 19472 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 35 ms (attempt = 1) W1017 23:32:43.543761 19472 catalog_manager.cc:3917] Failed to send 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:43.543848 19472 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 17 ms (attempt = 1) W1017 23:32:43.543900 19472 catalog_manager.cc:3917] Failed to send 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:43.544239 19472 catalog_manager.cc:3386] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1063e4b7405f470f859a82a5f2e0098a (attempt 1) I1017 23:32:43.549897 18839 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } unsafe_config_change: true I1017 23:32:43.550777 19522 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } unsafe_config_change: true } I1017 23:32:43.553980 19454 catalog_manager.cc:3301] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1063e4b7405f470f859a82a5f2e0098a with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I1017 23:32:43.554881 19472 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: config changed from index 3 to 5, NON_VOTER a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195) added. New cstate: current_term: 3 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I1017 23:32:43.557023 19472 catalog_manager.cc:3386] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 1063e4b7405f470f859a82a5f2e0098a (attempt 1) I1017 23:32:43.557979 18839 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195:46826): Closing peer: a0f0d49ea68149b9aef79c2d126bcc77 I1017 23:32:43.558127 18839 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } } unsafe_config_change: true I1017 23:32:43.559355 19523 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 57c44921a8064f9c84f3646d8158b003 (127.18.62.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } } unsafe_config_change: true } W1017 23:32:43.560878 18772 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195:46826): Couldn't send request to peer a0f0d49ea68149b9aef79c2d126bcc77 for tablet 1063e4b7405f470f859a82a5f2e0098a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1063e4b7405f470f859a82a5f2e0098a. Retrying in the next heartbeat period. Already tried 1 times. I1017 23:32:43.561370 19453 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 65 ms (attempt = 2) W1017 23:32:43.561461 19453 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:43.562341 19454 catalog_manager.cc:3301] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1063e4b7405f470f859a82a5f2e0098a with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I1017 23:32:43.563365 19472 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: config changed from index 5 to 6, NON_VOTER 57c44921a8064f9c84f3646d8158b003 (127.18.62.196) added. New cstate: current_term: 3 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W1017 23:32:43.565834 18771 consensus_peers.cc:447] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer 57c44921a8064f9c84f3646d8158b003 (127.18.62.196:49688): Couldn't send request to peer 57c44921a8064f9c84f3646d8158b003 for tablet 1063e4b7405f470f859a82a5f2e0098a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1063e4b7405f470f859a82a5f2e0098a. Retrying in the next heartbeat period. Already tried 1 times. I1017 23:32:43.579090 19455 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 51 ms (attempt = 2) W1017 23:32:43.579180 19455 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:43.626973 19453 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 97 ms (attempt = 3) W1017 23:32:43.627033 19453 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:43.630808 19455 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 107 ms (attempt = 3) W1017 23:32:43.630861 19455 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:43.724831 19453 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 132 ms (attempt = 4) W1017 23:32:43.724887 19453 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:43.738487 19455 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 134 ms (attempt = 4) W1017 23:32:43.738540 19455 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:43.857555 19453 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 287 ms (attempt = 5) W1017 23:32:43.857612 19453 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:43.873219 19455 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 281 ms (attempt = 5) W1017 23:32:43.873275 19455 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:43.949069 19536 ts_tablet_manager.cc:669] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Initiating tablet copy from peer 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193:53195) I1017 23:32:43.949295 19536 tablet_copy_client.cc:239] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: tablet copy: Beginning tablet copy session from remote peer at address 127.18.62.193:53195 I1017 23:32:43.955935 18859 tablet_copy_service.cc:135] P 92a7d6efbf034107a3f7e47dce492bb8: Received BeginTabletCopySession request for tablet 1063e4b7405f470f859a82a5f2e0098a from peer 57c44921a8064f9c84f3646d8158b003 ({username='jenkins'} at 127.18.62.196:45617) I1017 23:32:43.956061 18859 tablet_copy_service.cc:156] P 92a7d6efbf034107a3f7e47dce492bb8: Beginning new tablet copy session on tablet 1063e4b7405f470f859a82a5f2e0098a from peer 57c44921a8064f9c84f3646d8158b003 at {username='jenkins'} at 127.18.62.196:45617: session id = 57c44921a8064f9c84f3646d8158b003-1063e4b7405f470f859a82a5f2e0098a I1017 23:32:43.958057 18859 tablet_copy_source_session.cc:202] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Tablet Copy: opened 0 blocks and 1 log segments I1017 23:32:43.960801 19536 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1063e4b7405f470f859a82a5f2e0098a. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:43.971897 19536 ts_tablet_manager.cc:1173] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Registered tablet (data state: TABLET_DATA_COPYING) I1017 23:32:43.972167 19536 tablet_copy_client.cc:566] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: tablet copy: Starting download of 0 data blocks... I1017 23:32:43.972381 19536 tablet_copy_client.cc:529] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: tablet copy: Starting download of 1 WAL segments... I1017 23:32:43.974465 19536 tablet_copy_client.cc:405] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1017 23:32:43.976162 19536 ts_tablet_manager.cc:1028] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Bootstrapping tablet I1017 23:32:43.976374 19536 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Bootstrap starting. I1017 23:32:43.980665 19536 log.cc:525] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Log is configured to *not* fsync() on all Append() calls I1017 23:32:43.985543 19536 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1017 23:32:43.985854 19536 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Bootstrap complete. I1017 23:32:43.986057 19536 ts_tablet_manager.cc:1045] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s I1017 23:32:43.991075 19536 raft_consensus.cc:339] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } } unsafe_config_change: true I1017 23:32:43.991403 19536 raft_consensus.cc:704] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 57c44921a8064f9c84f3646d8158b003, State: Initialized, Role: LEARNER I1017 23:32:43.991658 19536 consensus_queue.cc:229] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } } unsafe_config_change: true I1017 23:32:43.993672 19536 ts_tablet_manager.cc:1073] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s I1017 23:32:43.994405 18859 tablet_copy_service.cc:325] P 92a7d6efbf034107a3f7e47dce492bb8: Request end of tablet copy session 57c44921a8064f9c84f3646d8158b003-1063e4b7405f470f859a82a5f2e0098a received from {username='jenkins'} at 127.18.62.196:45617 I1017 23:32:43.994490 18859 tablet_copy_service.cc:413] P 92a7d6efbf034107a3f7e47dce492bb8: ending tablet copy session 57c44921a8064f9c84f3646d8158b003-1063e4b7405f470f859a82a5f2e0098a on tablet 1063e4b7405f470f859a82a5f2e0098a with peer 57c44921a8064f9c84f3646d8158b003 I1017 23:32:44.139701 19540 ts_tablet_manager.cc:669] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Initiating tablet copy from peer 92a7d6efbf034107a3f7e47dce492bb8 (127.18.62.193:53195) I1017 23:32:44.139961 19540 tablet_copy_client.cc:239] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: tablet copy: Beginning tablet copy session from remote peer at address 127.18.62.193:53195 I1017 23:32:44.144291 18859 tablet_copy_service.cc:135] P 92a7d6efbf034107a3f7e47dce492bb8: Received BeginTabletCopySession request for tablet 1063e4b7405f470f859a82a5f2e0098a from peer a0f0d49ea68149b9aef79c2d126bcc77 ({username='jenkins'} at 127.18.62.195:54827) I1017 23:32:44.144388 18859 tablet_copy_service.cc:156] P 92a7d6efbf034107a3f7e47dce492bb8: Beginning new tablet copy session on tablet 1063e4b7405f470f859a82a5f2e0098a from peer a0f0d49ea68149b9aef79c2d126bcc77 at {username='jenkins'} at 127.18.62.195:54827: session id = a0f0d49ea68149b9aef79c2d126bcc77-1063e4b7405f470f859a82a5f2e0098a I1017 23:32:44.144795 19453 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 528 ms (attempt = 6) W1017 23:32:44.144862 19453 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:44.146157 18859 tablet_copy_source_session.cc:202] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: Tablet Copy: opened 0 blocks and 1 log segments I1017 23:32:44.148624 19540 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 1063e4b7405f470f859a82a5f2e0098a. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:44.154399 19455 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 523 ms (attempt = 6) W1017 23:32:44.154476 19455 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:44.158638 19540 ts_tablet_manager.cc:1173] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Registered tablet (data state: TABLET_DATA_COPYING) I1017 23:32:44.158887 19540 tablet_copy_client.cc:566] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: tablet copy: Starting download of 0 data blocks... I1017 23:32:44.159070 19540 tablet_copy_client.cc:529] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: tablet copy: Starting download of 1 WAL segments... I1017 23:32:44.160545 19540 tablet_copy_client.cc:405] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1017 23:32:44.162178 19540 ts_tablet_manager.cc:1028] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Bootstrapping tablet I1017 23:32:44.162381 19540 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Bootstrap starting. I1017 23:32:44.167284 19540 log.cc:525] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Log is configured to *not* fsync() on all Append() calls I1017 23:32:44.171996 19540 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1017 23:32:44.172332 19540 tablet_bootstrap.cc:438] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Bootstrap complete. I1017 23:32:44.172552 19540 ts_tablet_manager.cc:1045] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s I1017 23:32:44.177495 19540 raft_consensus.cc:339] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } } unsafe_config_change: true I1017 23:32:44.177816 19540 raft_consensus.cc:704] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a0f0d49ea68149b9aef79c2d126bcc77, State: Initialized, Role: LEARNER I1017 23:32:44.178069 19540 consensus_queue.cc:229] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: NON_VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: true } } unsafe_config_change: true I1017 23:32:44.179498 19540 ts_tablet_manager.cc:1073] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s I1017 23:32:44.180227 18859 tablet_copy_service.cc:325] P 92a7d6efbf034107a3f7e47dce492bb8: Request end of tablet copy session a0f0d49ea68149b9aef79c2d126bcc77-1063e4b7405f470f859a82a5f2e0098a received from {username='jenkins'} at 127.18.62.195:54827 I1017 23:32:44.180292 18859 tablet_copy_service.cc:413] P 92a7d6efbf034107a3f7e47dce492bb8: ending tablet copy session a0f0d49ea68149b9aef79c2d126bcc77-1063e4b7405f470f859a82a5f2e0098a on tablet 1063e4b7405f470f859a82a5f2e0098a with peer a0f0d49ea68149b9aef79c2d126bcc77 I1017 23:32:44.350718 19206 raft_consensus.cc:1125] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I1017 23:32:44.673715 19453 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af with a delay of 1041 ms (attempt = 7) W1017 23:32:44.673776 19453 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=2bd18a72ed7d447da0448d9193fa13af failed: Not found: failed to reset TS proxy: Could not find TS for UUID 2bd18a72ed7d447da0448d9193fa13af I1017 23:32:44.678360 19455 catalog_manager.cc:2938] Scheduling retry of 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 with a delay of 1036 ms (attempt = 7) W1017 23:32:44.678438 19455 catalog_manager.cc:2957] Async tablet task 1063e4b7405f470f859a82a5f2e0098a Delete Tablet RPC for TS=b8818ab3ec794a36a38bce58c4599155 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b8818ab3ec794a36a38bce58c4599155 I1017 23:32:44.700695 19084 raft_consensus.cc:1125] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I1017 23:32:44.892333 19545 raft_consensus.cc:973] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: attempting to promote NON_VOTER 57c44921a8064f9c84f3646d8158b003 to VOTER I1017 23:32:44.892653 19545 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer 57c44921a8064f9c84f3646d8158b003 (127.18.62.196:49688): Closing peer: 57c44921a8064f9c84f3646d8158b003 I1017 23:32:44.892724 19545 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195:46826): Closing peer: a0f0d49ea68149b9aef79c2d126bcc77 I1017 23:32:44.892958 19545 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true I1017 23:32:44.895043 19084 raft_consensus.cc:1180] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 LEARNER]: Refusing update from remote peer 92a7d6efbf034107a3f7e47dce492bb8: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I1017 23:32:44.895042 19206 raft_consensus.cc:1180] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 LEARNER]: Refusing update from remote peer 92a7d6efbf034107a3f7e47dce492bb8: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I1017 23:32:44.895977 19544 consensus_queue.cc:976] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1017 23:32:44.896214 19535 consensus_queue.cc:976] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1017 23:32:44.899432 19535 raft_consensus.cc:936] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: attempt to promote peer a0f0d49ea68149b9aef79c2d126bcc77: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I1017 23:32:44.899602 19544 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 57c44921a8064f9c84f3646d8158b003 (127.18.62.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true } I1017 23:32:44.900162 19206 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 57c44921a8064f9c84f3646d8158b003 (127.18.62.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true } I1017 23:32:44.903472 19084 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 57c44921a8064f9c84f3646d8158b003 (127.18.62.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true } I1017 23:32:44.904734 19472 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: config changed from index 6 to 7, 57c44921a8064f9c84f3646d8158b003 (127.18.62.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: NON_VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I1017 23:32:44.907479 19545 raft_consensus.cc:973] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8: attempting to promote NON_VOTER a0f0d49ea68149b9aef79c2d126bcc77 to VOTER I1017 23:32:44.907680 19545 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer 57c44921a8064f9c84f3646d8158b003 (127.18.62.196:49688): Closing peer: 57c44921a8064f9c84f3646d8158b003 I1017 23:32:44.907737 19545 consensus_peers.cc:469] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 -> Peer a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195:46826): Closing peer: a0f0d49ea68149b9aef79c2d126bcc77 I1017 23:32:44.907912 19545 consensus_queue.cc:206] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: false } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true I1017 23:32:44.909484 19206 raft_consensus.cc:1180] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 FOLLOWER]: Refusing update from remote peer 92a7d6efbf034107a3f7e47dce492bb8: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I1017 23:32:44.909528 19084 raft_consensus.cc:1180] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 LEARNER]: Refusing update from remote peer 92a7d6efbf034107a3f7e47dce492bb8: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I1017 23:32:44.910173 19544 consensus_queue.cc:976] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1017 23:32:44.910425 19535 consensus_queue.cc:976] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1017 23:32:44.912643 19544 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: false } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true } I1017 23:32:44.913998 19206 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P 57c44921a8064f9c84f3646d8158b003 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: false } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true } I1017 23:32:44.914327 19084 raft_consensus.cc:2794] T 1063e4b7405f470f859a82a5f2e0098a P a0f0d49ea68149b9aef79c2d126bcc77 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: false } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } } unsafe_config_change: true } I1017 23:32:44.917873 19472 catalog_manager.cc:3758] T 1063e4b7405f470f859a82a5f2e0098a P 92a7d6efbf034107a3f7e47dce492bb8 reported cstate change: config changed from index 7 to 8, a0f0d49ea68149b9aef79c2d126bcc77 (127.18.62.195) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "92a7d6efbf034107a3f7e47dce492bb8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 53195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 46826 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" member_type: VOTER last_known_addr { host: "127.18.62.196" port: 49688 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I1017 23:32:44.927640 18683 kudu-admin-test.cc:856] Waiting for Master to see new config... I1017 23:32:44.932823 18683 kudu-admin-test.cc:861] Tablet locations: tablet_id: "1063e4b7405f470f859a82a5f2e0098a" replicas { ts_info { permanent_uuid: "92a7d6efbf034107a3f7e47dce492bb8" rpc_addresses { host: "127.18.62.193" port: 53195 } } role: LEADER } replicas { ts_info { permanent_uuid: "a0f0d49ea68149b9aef79c2d126bcc77" rpc_addresses { host: "127.18.62.195" port: 46826 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "57c44921a8064f9c84f3646d8158b003" rpc_addresses { host: "127.18.62.196" port: 49688 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I1017 23:32:44.934823 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18753 I1017 23:32:44.935520 18879 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:44.986667 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18753 W1017 23:32:44.996397 19138 connection.cc:512] client connection to 127.18.62.193:53195 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.193:53195): Connection reset by peer (error 104) W1017 23:32:44.996417 19015 connection.cc:512] server connection from 127.18.62.193:38980 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.193:38980): Connection reset by peer (error 104) W1017 23:32:44.996409 19454 connection.cc:512] server connection from 127.18.62.193:44893 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.193:44893): Connection reset by peer (error 104) W1017 23:32:44.996922 19016 connection.cc:512] client connection to 127.18.62.193:53195 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.193:53195): Connection reset by peer (error 104) I1017 23:32:44.997215 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19007 W1017 23:32:44.997205 19140 connection.cc:512] server connection from 127.18.62.193:44736 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.193:44736): Connection reset by peer (error 104) I1017 23:32:44.997999 19124 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:45.043081 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19007 W1017 23:32:45.052096 19454 connection.cc:512] server connection from 127.18.62.195:37118 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.195:37118): Connection reset by peer (error 104) I1017 23:32:45.052238 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19130 I1017 23:32:45.053084 19246 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:45.099189 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19130 W1017 23:32:45.108261 19456 connection.cc:512] server connection from 127.18.62.196:51237 recv error: Network error: failed to read from TLS socket (remote: 127.18.62.196:51237): Connection reset by peer (error 104) I1017 23:32:45.108405 18683 external_mini_cluster.cc:1084] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 19445 I1017 23:32:45.109233 19502 generic_service.cc:169] Checking for leaks (request via RPC) I1017 23:32:45.172335 18683 external_mini_cluster.cc:1043] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 19445 [ OK ] AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig (8374 ms) [ RUN ] AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown I1017 23:32:45.197626 18683 test_util.cc:209] Using random seed: -194666194 I1017 23:32:45.199333 18683 ts_itest-base.cc:113] Starting cluster with: I1017 23:32:45.199367 18683 ts_itest-base.cc:114] -------------- I1017 23:32:45.199393 18683 ts_itest-base.cc:115] 3 tablet servers I1017 23:32:45.199419 18683 ts_itest-base.cc:116] 3 replicas per TS I1017 23:32:45.199445 18683 ts_itest-base.cc:117] -------------- I1017 23:32:45.200697 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.18.62.254:59590 --webserver_interface=127.18.62.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:45.391117 19566 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:45.391356 19566 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:45.393682 19566 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W1017 23:32:45.393730 19566 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:45.393779 19566 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W1017 23:32:45.393831 19566 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I1017 23:32:45.402691 19566 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.18.62.254:59590 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.18.62.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:45.409888 19566 master_main.cc:80] Initializing master server... I1017 23:32:45.411463 19566 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 750673us I1017 23:32:45.412617 19566 fs_manager.cc:263] Metadata directory not provided I1017 23:32:45.412683 19566 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:45.413347 19566 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:45.413419 19566 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:45.417062 19566 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/instance: uuid: "cb8da63688ef4b0c80eee0371eea5e2f" format_stamp: "Formatted at 2018-10-18 06:32:45 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:45.417623 19566 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "cb8da63688ef4b0c80eee0371eea5e2f" format_stamp: "Formatted at 2018-10-18 06:32:45 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:45.425736 19566 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s I1017 23:32:45.427464 19566 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:45.427877 19566 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:45.431710 19566 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I1017 23:32:45.431774 19566 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal uuid: "cb8da63688ef4b0c80eee0371eea5e2f" format_stamp: "Formatted at 2018-10-18 06:32:45 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:45.431885 19566 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:45.484452 19566 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:45.485924 19566 master_main.cc:83] Starting Master server... I1017 23:32:45.525257 19566 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.254:59590 I1017 23:32:45.525749 19566 webserver.cc:175] Starting webserver on 127.18.62.254:0 I1017 23:32:45.525794 19566 webserver.cc:186] Document root disabled I1017 23:32:45.526546 19566 webserver.cc:313] Webserver started. Bound to: http://127.18.62.254:49835/ I1017 23:32:45.527165 19566 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/master-0/data/info.pb I1017 23:32:45.531054 19625 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:45.531466 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 19566 I1017 23:32:45.532107 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.18.62.193:0 --local_ip_for_outbound_sockets=127.18.62.193 --webserver_interface=127.18.62.193 --webserver_port=0 --tserver_master_addrs=127.18.62.254:59590 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:45.543289 19625 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: Bootstrap starting. I1017 23:32:45.545665 19625 tablet_bootstrap.cc:592] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: No blocks or log segments found. Creating new log. I1017 23:32:45.546428 19625 log.cc:525] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: Log is configured to *not* fsync() on all Append() calls I1017 23:32:45.549199 19625 tablet_bootstrap.cc:438] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: No bootstrap required, opened a new log I1017 23:32:45.555644 19625 raft_consensus.cc:339] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } I1017 23:32:45.555876 19625 raft_consensus.cc:365] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:45.555946 19625 raft_consensus.cc:704] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8da63688ef4b0c80eee0371eea5e2f, State: Initialized, Role: FOLLOWER I1017 23:32:45.556429 19625 consensus_queue.cc:229] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } I1017 23:32:45.556638 19625 raft_consensus.cc:379] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1017 23:32:45.556725 19625 raft_consensus.cc:471] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1017 23:32:45.556812 19625 raft_consensus.cc:2840] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: Snoozing failure detection for 2.109s (starting election) I1017 23:32:45.556910 19625 raft_consensus.cc:2892] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:45.558375 19625 raft_consensus.cc:493] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } I1017 23:32:45.558799 19625 leader_election.cc:278] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1017 23:32:45.559834 19628 raft_consensus.cc:2840] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: Snoozing failure detection for 2.070s (election complete) I1017 23:32:45.560112 19628 raft_consensus.cc:2646] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 1 FOLLOWER]: Leader election won for term 1 I1017 23:32:45.562402 19628 raft_consensus.cc:666] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [term 1 LEADER]: Becoming Leader. State: Replica: cb8da63688ef4b0c80eee0371eea5e2f, State: Running, Role: LEADER I1017 23:32:45.562832 19628 consensus_queue.cc:206] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } I1017 23:32:45.565052 19625 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } } I1017 23:32:45.565912 19625 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: This master's current role is: LEADER I1017 23:32:45.564930 19629 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } } I1017 23:32:45.566289 19629 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: This master's current role is: LEADER I1017 23:32:45.565914 19630 sys_catalog.cc:337] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb8da63688ef4b0c80eee0371eea5e2f. Latest consensus state: current_term: 1 leader_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb8da63688ef4b0c80eee0371eea5e2f" member_type: VOTER } } I1017 23:32:45.570379 19630 sys_catalog.cc:340] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: This master's current role is: LEADER I1017 23:32:45.572536 19625 sys_catalog.cc:424] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f [sys.catalog]: configured and running, proceeding with master startup. I1017 23:32:45.573879 19566 master_main.cc:86] Master server successfully started. W1017 23:32:45.580724 19633 catalog_manager.cc:1084] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: acquiring CA information for follower catalog manager: Not found: root CA entry not found W1017 23:32:45.580832 19633 catalog_manager.cc:583] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I1017 23:32:45.580947 19632 catalog_manager.cc:1026] Loading table and tablet metadata into memory... I1017 23:32:45.584830 19632 catalog_manager.cc:1037] Initializing Kudu internal certificate authority... I1017 23:32:45.618772 19632 catalog_manager.cc:924] Generated new certificate authority record I1017 23:32:45.621258 19632 catalog_manager.cc:1046] Loading token signing keys... I1017 23:32:45.646359 19632 catalog_manager.cc:4062] T 00000000000000000000000000000000 P cb8da63688ef4b0c80eee0371eea5e2f: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:45.703778 19626 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:45.703994 19626 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:45.706027 19626 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:45.706137 19626 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.193 I1017 23:32:45.714030 19626 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.18.62.193 --webserver_port=0 --tserver_master_addrs=127.18.62.254:59590 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:45.722380 19626 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:45.723593 19626 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 750673us I1017 23:32:45.724625 19626 fs_manager.cc:263] Metadata directory not provided I1017 23:32:45.724690 19626 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:45.725347 19626 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:45.725433 19626 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:45.728668 19626 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "0ecfe08279804cb0888b118fc4f87243" format_stamp: "Formatted at 2018-10-18 06:32:45 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:45.729202 19626 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "0ecfe08279804cb0888b118fc4f87243" format_stamp: "Formatted at 2018-10-18 06:32:45 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:45.737685 19626 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s I1017 23:32:45.738957 19626 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:45.739365 19626 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:45.744145 19626 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s I1017 23:32:45.744208 19626 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal uuid: "0ecfe08279804cb0888b118fc4f87243" format_stamp: "Formatted at 2018-10-18 06:32:45 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:45.744333 19626 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:45.786651 19626 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:45.788265 19626 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:45.788688 19626 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:45.892407 19626 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.193:39726 I1017 23:32:45.892928 19626 webserver.cc:175] Starting webserver on 127.18.62.193:0 I1017 23:32:45.892967 19626 webserver.cc:186] Document root disabled I1017 23:32:45.894435 19626 webserver.cc:313] Webserver started. Bound to: http://127.18.62.193:41689/ I1017 23:32:45.895021 19626 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-0/data/info.pb I1017 23:32:45.897277 19626 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:45.904072 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19626 I1017 23:32:45.904774 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.18.62.194:0 --local_ip_for_outbound_sockets=127.18.62.194 --webserver_interface=127.18.62.194 --webserver_port=0 --tserver_master_addrs=127.18.62.254:59590 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:45.927506 19755 heartbeater.cc:346] Connected to a master server at 127.18.62.254:59590 I1017 23:32:45.927703 19755 heartbeater.cc:426] Registering TS with master... I1017 23:32:45.928745 19755 heartbeater.cc:475] Master 127.18.62.254:59590 requested a full tablet report, sending... I1017 23:32:45.930833 19592 ts_manager.cc:98] Registered new tserver with Master: 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:45.934195 19592 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.193:39759 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:46.081065 19759 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:46.081316 19759 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:46.083345 19759 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:46.083483 19759 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.194 I1017 23:32:46.090445 19759 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.18.62.194 --webserver_port=0 --tserver_master_addrs=127.18.62.254:59590 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:46.098772 19759 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:46.099967 19759 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 751173us I1017 23:32:46.100989 19759 fs_manager.cc:263] Metadata directory not provided I1017 23:32:46.101056 19759 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:46.101763 19759 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:46.101830 19759 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:46.104962 19759 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b01d528fd3c74eb5b42b8d4888591ed2" format_stamp: "Formatted at 2018-10-18 06:32:46 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:46.105566 19759 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b01d528fd3c74eb5b42b8d4888591ed2" format_stamp: "Formatted at 2018-10-18 06:32:46 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:46.113556 19759 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.001s sys 0.005s I1017 23:32:46.114838 19759 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:46.115248 19759 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:46.119410 19759 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I1017 23:32:46.119473 19759 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal uuid: "b01d528fd3c74eb5b42b8d4888591ed2" format_stamp: "Formatted at 2018-10-18 06:32:46 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:46.119609 19759 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:46.162113 19759 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:46.164103 19759 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:46.164501 19759 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:46.256381 19759 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.194:38185 I1017 23:32:46.256851 19759 webserver.cc:175] Starting webserver on 127.18.62.194:0 I1017 23:32:46.256887 19759 webserver.cc:186] Document root disabled I1017 23:32:46.257645 19759 webserver.cc:313] Webserver started. Bound to: http://127.18.62.194:47318/ I1017 23:32:46.258227 19759 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-1/data/info.pb I1017 23:32:46.260138 19759 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:46.268121 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19759 I1017 23:32:46.268728 18683 external_mini_cluster.cc:845] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.18.62.195:0 --local_ip_for_outbound_sockets=127.18.62.195 --webserver_interface=127.18.62.195 --webserver_port=0 --tserver_master_addrs=127.18.62.254:59590 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1017 23:32:46.275929 19882 heartbeater.cc:346] Connected to a master server at 127.18.62.254:59590 I1017 23:32:46.276078 19882 heartbeater.cc:426] Registering TS with master... I1017 23:32:46.276643 19882 heartbeater.cc:475] Master 127.18.62.254:59590 requested a full tablet report, sending... I1017 23:32:46.277721 19592 ts_manager.cc:98] Registered new tserver with Master: b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:46.279063 19592 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.194:55568 WARNING: Logging before InitGoogleLogging() is written to STDERR W1017 23:32:46.434903 19885 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1017 23:32:46.435118 19885 flags.cc:406] Enabled unsafe flag: --never_fsync=true W1017 23:32:46.437083 19885 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W1017 23:32:46.437172 19885 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.62.195 I1017 23:32:46.444169 19885 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.18.62.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.18.62.195 --webserver_port=0 --tserver_master_addrs=127.18.62.254:59590 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.18.62.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.9.0-SNAPSHOT revision 7a4caf9660617c8dad2f95b939adec9eb1f9fd04 build type FASTDEBUG built by jenkins at 17 Oct 2018 22:26:55 PST on quasar-lixwuo-1.vpc.cloudera.com build id 166828 ASAN enabled I1017 23:32:46.451840 19885 tablet_server_main.cc:85] Initializing tablet server... I1017 23:32:46.453018 19885 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 751173us I1017 23:32:46.454041 19885 fs_manager.cc:263] Metadata directory not provided I1017 23:32:46.454097 19885 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I1017 23:32:46.454751 19885 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I1017 23:32:46.454813 19885 server_base.cc:433] Attempting to create new FS layout instead I1017 23:32:46.457921 19885 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "1ad000b8f6704a80bc4330f4e0c41996" format_stamp: "Formatted at 2018-10-18 06:32:46 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:46.458442 19885 fs_manager.cc:602] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "1ad000b8f6704a80bc4330f4e0c41996" format_stamp: "Formatted at 2018-10-18 06:32:46 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:46.466804 19885 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I1017 23:32:46.468065 19885 env_posix.cc:1676] Not raising this process' open files per process limit of 32768; it is already as high as it can go I1017 23:32:46.468493 19885 file_cache.cc:466] Constructed file cache lbm with capacity 13107 I1017 23:32:46.472141 19885 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I1017 23:32:46.472203 19885 fs_manager.cc:436] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal uuid: "1ad000b8f6704a80bc4330f4e0c41996" format_stamp: "Formatted at 2018-10-18 06:32:46 on quasar-lixwuo-1.vpc.cloudera.com" I1017 23:32:46.472321 19885 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1017 23:32:46.528708 19885 env_posix.cc:1671] Raising this process' running threads per effective uid limit from 32768 to 65536 I1017 23:32:46.530683 19885 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I1017 23:32:46.531049 19885 tablet_server_main.cc:90] Starting tablet server... I1017 23:32:46.621618 19885 rpc_server.cc:205] RPC server started. Bound to: 127.18.62.195:55672 I1017 23:32:46.622058 19885 webserver.cc:175] Starting webserver on 127.18.62.195:0 I1017 23:32:46.622094 19885 webserver.cc:186] Document root disabled I1017 23:32:46.623317 19885 webserver.cc:313] Webserver started. Bound to: http://127.18.62.195:56449/ I1017 23:32:46.623914 19885 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.5.AdminCliTest.TestSimultaneousLeaderTransferAndAbruptStepdown.1539844356806883-18683/raft_consensus-itest-cluster/ts-2/data/info.pb I1017 23:32:46.626520 19885 tablet_server_main.cc:93] Tablet server successfully started. I1017 23:32:46.630153 18683 external_mini_cluster.cc:907] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19885 I1017 23:32:46.638694 20004 heartbeater.cc:346] Connected to a master server at 127.18.62.254:59590 I1017 23:32:46.638825 20004 heartbeater.cc:426] Registering TS with master... I1017 23:32:46.639737 20004 heartbeater.cc:475] Master 127.18.62.254:59590 requested a full tablet report, sending... I1017 23:32:46.640976 19591 ts_manager.cc:98] Registered new tserver with Master: 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:46.642079 19591 master_service.cc:253] Signed X509 certificate for tserver {username='jenkins'} at 127.18.62.195:57449 I1017 23:32:46.642623 18683 external_mini_cluster.cc:535] 3 TS(s) registered with all masters I1017 23:32:46.652412 19591 catalog_manager.cc:1366] Servicing CreateTable request from {username='jenkins'} at 127.18.62.254:59277: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W1017 23:32:46.657253 19591 catalog_manager.cc:1504] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I1017 23:32:46.677790 19692 tablet_service.cc:765] Processing CreateTablet for tablet 9b4b2dea960941bcb38197b51c55baf4 (table=TestTable [id=e9b0e25393464dc3965d604c5cff7330]), partition=RANGE (key) PARTITION UNBOUNDED I1017 23:32:46.678380 19941 tablet_service.cc:765] Processing CreateTablet for tablet 9b4b2dea960941bcb38197b51c55baf4 (table=TestTable [id=e9b0e25393464dc3965d604c5cff7330]), partition=RANGE (key) PARTITION UNBOUNDED I1017 23:32:46.678676 19692 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9b4b2dea960941bcb38197b51c55baf4. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:46.678988 19941 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9b4b2dea960941bcb38197b51c55baf4. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:46.684281 19819 tablet_service.cc:765] Processing CreateTablet for tablet 9b4b2dea960941bcb38197b51c55baf4 (table=TestTable [id=e9b0e25393464dc3965d604c5cff7330]), partition=RANGE (key) PARTITION UNBOUNDED I1017 23:32:46.685079 19819 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 9b4b2dea960941bcb38197b51c55baf4. 1 dirs total, 0 dirs full, 0 dirs failed I1017 23:32:46.690623 19692 ts_tablet_manager.cc:1173] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Registered tablet (data state: TABLET_DATA_READY) I1017 23:32:46.691534 20024 ts_tablet_manager.cc:1028] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Bootstrapping tablet I1017 23:32:46.692137 20024 tablet_bootstrap.cc:438] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Bootstrap starting. I1017 23:32:46.692502 19941 ts_tablet_manager.cc:1173] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Registered tablet (data state: TABLET_DATA_READY) I1017 23:32:46.694064 20025 ts_tablet_manager.cc:1028] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Bootstrapping tablet I1017 23:32:46.694473 20025 tablet_bootstrap.cc:438] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Bootstrap starting. I1017 23:32:46.696219 20024 tablet_bootstrap.cc:592] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: No blocks or log segments found. Creating new log. I1017 23:32:46.698140 20024 log.cc:525] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Log is configured to *not* fsync() on all Append() calls I1017 23:32:46.698289 19819 ts_tablet_manager.cc:1173] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Registered tablet (data state: TABLET_DATA_READY) I1017 23:32:46.698316 20025 tablet_bootstrap.cc:592] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: No blocks or log segments found. Creating new log. I1017 23:32:46.699151 20025 log.cc:525] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Log is configured to *not* fsync() on all Append() calls I1017 23:32:46.699832 20027 ts_tablet_manager.cc:1028] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Bootstrapping tablet I1017 23:32:46.700151 20027 tablet_bootstrap.cc:438] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Bootstrap starting. I1017 23:32:46.702796 20027 tablet_bootstrap.cc:592] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: No blocks or log segments found. Creating new log. I1017 23:32:46.703490 20027 log.cc:525] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Log is configured to *not* fsync() on all Append() calls I1017 23:32:46.704825 20024 tablet_bootstrap.cc:438] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: No bootstrap required, opened a new log I1017 23:32:46.705049 20025 tablet_bootstrap.cc:438] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: No bootstrap required, opened a new log I1017 23:32:46.705054 20024 ts_tablet_manager.cc:1045] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s I1017 23:32:46.705246 20025 ts_tablet_manager.cc:1045] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s I1017 23:32:46.706627 20027 tablet_bootstrap.cc:438] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: No bootstrap required, opened a new log I1017 23:32:46.706789 20027 ts_tablet_manager.cc:1045] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s I1017 23:32:46.713084 20027 raft_consensus.cc:339] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.713552 20027 raft_consensus.cc:365] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:46.713722 20027 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01d528fd3c74eb5b42b8d4888591ed2, State: Initialized, Role: FOLLOWER I1017 23:32:46.714164 20027 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.714170 20024 raft_consensus.cc:339] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.714623 20024 raft_consensus.cc:365] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:46.714357 20025 raft_consensus.cc:339] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.714726 20024 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Initialized, Role: FOLLOWER I1017 23:32:46.714787 20025 raft_consensus.cc:365] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1017 23:32:46.714890 20025 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Initialized, Role: FOLLOWER I1017 23:32:46.715240 20024 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.715715 19882 heartbeater.cc:467] Master 127.18.62.254:59590 was elected leader, sending a full tablet report... I1017 23:32:46.715301 20025 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.716895 20027 ts_tablet_manager.cc:1073] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s I1017 23:32:46.717265 19755 heartbeater.cc:467] Master 127.18.62.254:59590 was elected leader, sending a full tablet report... I1017 23:32:46.717698 20024 ts_tablet_manager.cc:1073] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s I1017 23:32:46.719143 20004 heartbeater.cc:467] Master 127.18.62.254:59590 was elected leader, sending a full tablet report... I1017 23:32:46.719781 20025 ts_tablet_manager.cc:1073] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s W1017 23:32:46.760478 19883 tablet.cc:1744] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Can't schedule compaction. Clean time has not been advanced past its initial value. I1017 23:32:46.873937 20032 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:46.874449 20032 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.665s (starting election) I1017 23:32:46.874562 20032 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.875721 20032 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:46.876044 20032 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) W1017 23:32:46.876981 20005 tablet.cc:1744] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Can't schedule compaction. Clean time has not been advanced past its initial value. I1017 23:32:46.882231 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:32:46.883021 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.085s (vote granted) I1017 23:32:46.883128 19712 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 0. I1017 23:32:46.883494 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:46.883704 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.840s (vote granted) I1017 23:32:46.883719 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:46.883791 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I1017 23:32:46.883785 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 0. I1017 23:32:46.884066 20032 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.951s (election complete) I1017 23:32:46.884169 20032 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1017 23:32:46.884306 20032 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:46.884310 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:46.884413 20032 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.680s (starting election) I1017 23:32:46.884506 20032 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:46.886008 20032 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.886343 20032 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:46.886557 20032 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:46.887051 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:32:46.887166 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:46.887257 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.198s (vote granted) I1017 23:32:46.887228 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:46.887338 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 0 FOLLOWER]: Advancing to term 1 I1017 23:32:46.887414 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.583s (vote granted) I1017 23:32:46.888685 19712 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 1. I1017 23:32:46.888792 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 1. I1017 23:32:46.889130 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 election: Vote granted by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:46.889189 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1017 23:32:46.889365 20032 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.902s (election complete) I1017 23:32:46.889509 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 1 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:46.889511 20032 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 FOLLOWER]: Leader election won for term 1 I1017 23:32:46.890446 20032 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:46.890900 20032 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:46.894479 19592 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 0 to 1, leader changed from to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 1 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:32:46.898089 19756 tablet.cc:1744] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Can't schedule compaction. Clean time has not been advanced past its initial value. I1017 23:32:46.900466 18683 external_mini_cluster.cc:535] 3 TS(s) registered with all masters I1017 23:32:46.901643 18683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0ecfe08279804cb0888b118fc4f87243 to finish bootstrapping I1017 23:32:46.907099 18683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b01d528fd3c74eb5b42b8d4888591ed2 to finish bootstrapping I1017 23:32:46.911895 18683 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1ad000b8f6704a80bc4330f4e0c41996 to finish bootstrapping I1017 23:32:46.930219 18683 test_util.cc:209] Using random seed: -192933634 I1017 23:32:46.942847 18683 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I1017 23:32:46.967241 19839 raft_consensus.cc:1180] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 1 FOLLOWER]: Refusing update from remote peer 1ad000b8f6704a80bc4330f4e0c41996: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1017 23:32:46.967371 19712 raft_consensus.cc:1180] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 1 FOLLOWER]: Refusing update from remote peer 1ad000b8f6704a80bc4330f4e0c41996: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1017 23:32:46.968605 20034 consensus_queue.cc:976] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1017 23:32:46.969020 20032 consensus_queue.cc:976] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W1017 23:32:47.217578 20043 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:47.302721 19961 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40009 I1017 23:32:47.302925 19961 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 LEADER]: Received request to transfer leadership I1017 23:32:47.303627 20034 raft_consensus.cc:902] Instructing follower b01d528fd3c74eb5b42b8d4888591ed2 to start an election I1017 23:32:47.303843 20032 raft_consensus.cc:990] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 LEADER]: Signalling peer b01d528fd3c74eb5b42b8d4888591ed2to start an election I1017 23:32:47.306358 19839 tablet_service.cc:1118] Received Run Leader Election RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" from {username='jenkins'} at 127.18.62.195:35243 I1017 23:32:47.306865 19839 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1017 23:32:47.306980 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.235s (starting election) I1017 23:32:47.307082 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 1 FOLLOWER]: Advancing to term 2 I1017 23:32:47.309052 19839 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:47.309772 19839 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 2 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:47.310036 19839 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 2 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:47.321497 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" candidate_term: 2 candidate_status { last_received { term: 1 index: 56 } } ignore_live_leader: true dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:32:47.321473 19961 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" candidate_term: 2 candidate_status { last_received { term: 1 index: 56 } } ignore_live_leader: true dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:32:47.321640 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 1 FOLLOWER]: Advancing to term 2 I1017 23:32:47.321650 19961 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 LEADER]: Stepping down as leader of term 1 I1017 23:32:47.321689 19961 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:47.321743 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.071s (vote granted) I1017 23:32:47.321920 19961 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:47.322100 19961 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:47.322176 19961 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:32:47.322368 19961 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 1 FOLLOWER]: Advancing to term 2 I1017 23:32:47.322455 19961 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.595s (vote granted) I1017 23:32:47.323688 19712 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01d528fd3c74eb5b42b8d4888591ed2 in term 2. I1017 23:32:47.324234 19767 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 2 election: Vote granted by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:47.324314 19767 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I1017 23:32:47.324592 19961 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b01d528fd3c74eb5b42b8d4888591ed2 in term 2. I1017 23:32:47.324671 20030 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.981s (election complete) I1017 23:32:47.324760 20030 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 FOLLOWER]: Leader election won for term 2 I1017 23:32:47.325182 19769 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 2 election: Vote granted by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:47.325668 20030 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 LEADER]: Becoming Leader. State: Replica: b01d528fd3c74eb5b42b8d4888591ed2, State: Running, Role: LEADER I1017 23:32:47.326105 20030 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 56, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:47.328711 19592 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 reported cstate change: term changed from 1 to 2, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194). New cstate: current_term: 2 leader_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } I1017 23:32:47.357599 19712 raft_consensus.cc:1180] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Refusing update from remote peer b01d528fd3c74eb5b42b8d4888591ed2: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 2 index: 58. (index mismatch) I1017 23:32:47.357900 19961 raft_consensus.cc:1180] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Refusing update from remote peer b01d528fd3c74eb5b42b8d4888591ed2: Log matching property violated. Preceding OpId in replica: term: 1 index: 56. Preceding OpId from leader: term: 2 index: 58. (index mismatch) I1017 23:32:47.358268 20072 consensus_queue.cc:976] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s I1017 23:32:47.358734 20030 consensus_queue.cc:976] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 57, Last known committed idx: 56, Time since last communication: 0.000s I1017 23:32:47.365633 20045 mvcc.cc:203] Tried to move safe_time back from 6307202528689979392 to 6307202528571133952. Current Snapshot: MvccSnapshot[committed={T|T < 6307202528689979392}] W1017 23:32:47.667241 20074 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:47.771051 19839 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" mode: ABRUPT from {username='jenkins'} at 127.18.62.194:54111 I1017 23:32:47.771333 19839 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 LEADER]: Received request to step down I1017 23:32:47.771438 19839 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 LEADER]: Stepping down as leader of term 2 I1017 23:32:47.771554 19839 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b01d528fd3c74eb5b42b8d4888591ed2, State: Running, Role: LEADER I1017 23:32:47.771792 19839 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.130, Last appended by leader: 130, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:47.772035 19839 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:32:47.772143 19839 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:32:47.772213 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 2 FOLLOWER]: Advancing to term 3 I1017 23:32:47.772282 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:32:49.056898 20091 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:49.115844 19839 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" mode: GRACEFUL from {username='jenkins'} at 127.18.62.194:54113 I1017 23:32:49.116012 19839 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Received request to transfer leadership I1017 23:32:49.116080 19839 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader I1017 23:32:49.273412 20110 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b01d528fd3c74eb5b42b8d4888591ed2) I1017 23:32:49.273537 20110 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.660s (starting election) I1017 23:32:49.273669 20110 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:49.274415 20110 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.274773 20110 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:49.286393 20111 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b01d528fd3c74eb5b42b8d4888591ed2) I1017 23:32:49.286512 20111 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.689s (starting election) I1017 23:32:49.286542 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:49.286607 20111 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:49.286723 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.572s (vote granted) I1017 23:32:49.286792 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 3. I1017 23:32:49.287058 20111 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:49.287297 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.287371 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I1017 23:32:49.287698 20110 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.186s (election complete) I1017 23:32:49.287796 20110 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Leader pre-election won for term 3 I1017 23:32:49.287894 20110 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b01d528fd3c74eb5b42b8d4888591ed2) I1017 23:32:49.287959 20110 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.989s (starting election) I1017 23:32:49.288019 20110 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 2 FOLLOWER]: Advancing to term 3 I1017 23:32:49.288436 20111 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.288993 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:32:49.289201 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:49.289299 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.040s (vote granted) I1017 23:32:49.289355 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 3. I1017 23:32:49.289844 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.289904 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I1017 23:32:49.290202 20111 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.667s (election complete) I1017 23:32:49.290331 20111 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Leader pre-election won for term 3 I1017 23:32:49.290405 20111 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b01d528fd3c74eb5b42b8d4888591ed2) I1017 23:32:49.290509 20111 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.986s (starting election) I1017 23:32:49.290591 20111 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 2 FOLLOWER]: Advancing to term 3 I1017 23:32:49.291247 20110 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:49.291673 19712 raft_consensus.cc:2282] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 in current term 3: Already voted for candidate 0ecfe08279804cb0888b118fc4f87243 in this term. I1017 23:32:49.292357 20110 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.292595 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 in current term 3: Already voted for candidate 0ecfe08279804cb0888b118fc4f87243 in this term. I1017 23:32:49.292897 20110 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:49.293131 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:49.293262 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.185s (vote granted) I1017 23:32:49.293488 20111 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:49.293992 20111 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:49.294785 20111 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.294854 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:32:49.295020 19712 raft_consensus.cc:2282] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 in current term 3: Already voted for candidate 0ecfe08279804cb0888b118fc4f87243 in this term. I1017 23:32:49.295212 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 3. I1017 23:32:49.295756 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:49.295855 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:49.295917 19839 raft_consensus.cc:2282] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 in current term 3: Already voted for candidate 0ecfe08279804cb0888b118fc4f87243 in this term. I1017 23:32:49.295917 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I1017 23:32:49.296012 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 in current term 3: Already voted for candidate 0ecfe08279804cb0888b118fc4f87243 in this term. I1017 23:32:49.296092 20110 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.092s (election complete) I1017 23:32:49.296177 20110 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Leader election won for term 3 I1017 23:32:49.296528 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 election: Vote denied by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 in current term 3: Already voted for candidate 0ecfe08279804cb0888b118fc4f87243 in this term. I1017 23:32:49.296689 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. I1017 23:32:49.296936 20110 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:49.298200 20110 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.130, Last appended by leader: 130, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:49.298678 19961 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:32:49.299202 19961 raft_consensus.cc:2282] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 in current term 3: Already voted for candidate 1ad000b8f6704a80bc4330f4e0c41996 in this term. I1017 23:32:49.299015 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 3 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:32:49.299445 19960 raft_consensus.cc:2282] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 in current term 3: Already voted for candidate 1ad000b8f6704a80bc4330f4e0c41996 in this term. I1017 23:32:49.300168 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 in current term 3: Already voted for candidate 1ad000b8f6704a80bc4330f4e0c41996 in this term. I1017 23:32:49.300581 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 3 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 in current term 3: Already voted for candidate 1ad000b8f6704a80bc4330f4e0c41996 in this term. I1017 23:32:49.300746 20111 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.719s (election complete) I1017 23:32:49.300822 20111 raft_consensus.cc:2594] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority I1017 23:32:49.302907 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 2 to 3, leader changed from b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 3 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:32:49.398970 20109 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:49.458225 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35275 I1017 23:32:49.458422 19712 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 LEADER]: Received request to step down I1017 23:32:49.458497 19712 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 LEADER]: Stepping down as leader of term 3 I1017 23:32:49.458534 19712 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:49.458746 19712 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 3.131, Last appended by leader: 131, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:49.458925 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:32:49.458982 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:49.459106 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 3 FOLLOWER]: Advancing to term 4 I1017 23:32:49.459165 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:32:50.731307 20138 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:50.791055 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35277 I1017 23:32:50.791198 19712 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Received request to transfer leadership I1017 23:32:50.791261 19712 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader I1017 23:32:51.021116 20157 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:51.021211 20157 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.730s (starting election) I1017 23:32:51.021291 20157 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:51.021723 20157 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:51.021883 20157 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:51.022855 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 4 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:32:51.023144 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 4 because replica has last-logged OpId of term: 3 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:51.023072 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 4 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:51.023239 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.917s (vote granted) I1017 23:32:51.023319 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 3. I1017 23:32:51.023893 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 4 because replica has last-logged OpId of term: 3 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:51.024066 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:51.024129 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. I1017 23:32:51.024329 20157 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.653s (election complete) I1017 23:32:51.024426 20157 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Leader pre-election won for term 4 I1017 23:32:51.024493 20157 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:51.024554 20157 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.731s (starting election) I1017 23:32:51.024617 20157 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 3 FOLLOWER]: Advancing to term 4 I1017 23:32:51.026054 20157 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:51.026371 20157 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:51.026568 20157 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:51.027135 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 4 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:32:51.027330 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 4 because replica has last-logged OpId of term: 3 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:51.027395 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 4 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:51.027551 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 3 FOLLOWER]: Advancing to term 4 I1017 23:32:51.027623 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.554s (vote granted) I1017 23:32:51.028026 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 4 because replica has last-logged OpId of term: 3 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:51.029362 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 4. I1017 23:32:51.029860 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:51.029917 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. I1017 23:32:51.030153 20157 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.881s (election complete) I1017 23:32:51.030226 20157 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 FOLLOWER]: Leader election won for term 4 I1017 23:32:51.030601 20157 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:51.031090 20157 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.130, Last appended by leader: 130, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:51.033547 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 3 to 4, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 4 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:32:51.064981 20156 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:51.124891 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40021 I1017 23:32:51.125058 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 LEADER]: Received request to step down I1017 23:32:51.125109 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 LEADER]: Stepping down as leader of term 4 I1017 23:32:51.125140 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:51.125352 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 4.131, Last appended by leader: 131, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:51.125525 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:51.125573 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:32:51.125653 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 4 FOLLOWER]: Advancing to term 5 I1017 23:32:51.125715 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:32:52.398273 20184 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:52.457257 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40024 I1017 23:32:52.457403 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Received request to transfer leadership I1017 23:32:52.457478 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:32:52.460590 20201 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:52.460664 20201 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.222s (starting election) I1017 23:32:52.460757 20201 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:52.461133 20201 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:52.461336 20201 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:52.462110 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 5 candidate_status { last_received { term: 3 index: 131 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:52.462255 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.059s (vote granted) I1017 23:32:52.462317 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 4. I1017 23:32:52.462602 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 5 candidate_status { last_received { term: 3 index: 131 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:32:52.462819 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 5 because replica has last-logged OpId of term: 4 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 131. I1017 23:32:52.462848 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:52.462905 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Illegal state: not currently leaderI1017 23:32:52.463089 20201 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.011s (election complete) I1017 23:32:52.463193 20201 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Leader pre-election won for term 5 I1017 23:32:52.463268 20201 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:52.463397 20201 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.067s (starting election) I1017 23:32:52.463415 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 5 because replica has last-logged OpId of term: 4 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 131. I1017 23:32:52.463543 20201 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 4 FOLLOWER]: Advancing to term 5 I1017 23:32:52.464977 20201 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:52.465286 20201 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:52.465498 20201 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:52.466084 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 5 candidate_status { last_received { term: 3 index: 131 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:52.466184 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 4 FOLLOWER]: Advancing to term 5 I1017 23:32:52.466189 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 5 candidate_status { last_received { term: 3 index: 131 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:32:52.466243 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.231s (vote granted) I1017 23:32:52.466374 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 5 because replica has last-logged OpId of term: 4 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 131. I1017 23:32:52.466817 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 5 because replica has last-logged OpId of term: 4 index: 131, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 131. I1017 23:32:52.467636 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 5. I1017 23:32:52.468067 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:52.468123 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. I1017 23:32:52.468307 20201 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.155s (election complete) I1017 23:32:52.468441 20201 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 FOLLOWER]: Leader election won for term 5 I1017 23:32:52.468772 20201 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:52.469141 20201 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 3.131, Last appended by leader: 131, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:52.471333 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 4 to 5, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 5 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:32:52.730588 20209 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:52.793870 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35284 I1017 23:32:52.794010 19712 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 LEADER]: Received request to step down I1017 23:32:52.794054 19712 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 LEADER]: Stepping down as leader of term 5 I1017 23:32:52.794083 19712 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:52.794276 19712 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 5.132, Last appended by leader: 132, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:52.794426 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:32:52.794467 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:52.794558 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 5 FOLLOWER]: Advancing to term 6 I1017 23:32:52.794620 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:32:54.067111 20226 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:54.125923 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35286 I1017 23:32:54.126063 19712 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Received request to transfer leadership I1017 23:32:54.126121 19712 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:32:54.126751 20243 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:54.126833 20243 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.079s (starting election) I1017 23:32:54.126960 20243 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:54.127725 20243 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:54.127926 20243 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:54.128487 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 6 candidate_status { last_received { term: 4 index: 131 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:32:54.128679 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 6 because replica has last-logged OpId of term: 5 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 131. I1017 23:32:54.128849 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 6 candidate_status { last_received { term: 4 index: 131 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:54.129051 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.058s (vote granted) I1017 23:32:54.129132 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 5. I1017 23:32:54.129660 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:54.129721 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. I1017 23:32:54.130105 20243 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.950s (election complete) I1017 23:32:54.130105 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 6 because replica has last-logged OpId of term: 5 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 131. I1017 23:32:54.130241 20243 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Leader pre-election won for term 6 I1017 23:32:54.130321 20243 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:54.130401 20243 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.791s (starting election) I1017 23:32:54.130472 20243 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 5 FOLLOWER]: Advancing to term 6 Illegal state: not currently leader I1017 23:32:54.131974 20243 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:54.132284 20243 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:54.132486 20243 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:54.132999 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 6 candidate_status { last_received { term: 4 index: 131 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:32:54.133215 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 6 because replica has last-logged OpId of term: 5 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 131. I1017 23:32:54.133165 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 6 candidate_status { last_received { term: 4 index: 131 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:54.133291 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 5 FOLLOWER]: Advancing to term 6 I1017 23:32:54.133361 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.202s (vote granted) I1017 23:32:54.133709 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 6 because replica has last-logged OpId of term: 5 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 131. I1017 23:32:54.134817 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 6. I1017 23:32:54.135249 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:54.135304 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. I1017 23:32:54.135555 20243 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.245s (election complete) I1017 23:32:54.135643 20243 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 FOLLOWER]: Leader election won for term 6 I1017 23:32:54.140282 20243 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:54.140678 20243 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 4.131, Last appended by leader: 131, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:54.143091 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 5 to 6, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 6 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:32:54.404503 20251 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:54.463863 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40030 I1017 23:32:54.464020 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 LEADER]: Received request to step down I1017 23:32:54.464082 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 LEADER]: Stepping down as leader of term 6 I1017 23:32:54.464128 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:54.464361 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 6.132, Last appended by leader: 132, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:54.464537 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:54.464591 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:32:54.464660 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 6 FOLLOWER]: Advancing to term 7 I1017 23:32:54.464715 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:32:55.742453 20268 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:55.795742 20283 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:55.795872 20283 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.700s (starting election) I1017 23:32:55.795964 20283 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:55.796365 20283 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:55.796547 20283 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:55.797668 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 7 candidate_status { last_received { term: 5 index: 132 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:55.797736 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 7 candidate_status { last_received { term: 5 index: 132 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:32:55.797857 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.898s (vote granted) I1017 23:32:55.797958 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 7 because replica has last-logged OpId of term: 6 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 132. I1017 23:32:55.797935 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 6. I1017 23:32:55.798625 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:55.798727 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I1017 23:32:55.798830 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 7 because replica has last-logged OpId of term: 6 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 132. I1017 23:32:55.799036 20283 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.980s (election complete) I1017 23:32:55.799139 20283 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Leader pre-election won for term 7 I1017 23:32:55.799181 20283 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:55.799216 20283 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.797s (starting election) I1017 23:32:55.799255 20283 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 6 FOLLOWER]: Advancing to term 7 I1017 23:32:55.800637 20283 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:55.800935 20283 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:55.801056 20283 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:55.801751 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 7 candidate_status { last_received { term: 5 index: 132 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:55.801874 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 6 FOLLOWER]: Advancing to term 7 I1017 23:32:55.801952 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.977s (vote granted) I1017 23:32:55.801877 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 7 candidate_status { last_received { term: 5 index: 132 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:32:55.802184 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 7 because replica has last-logged OpId of term: 6 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 132. I1017 23:32:55.802776 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 7 because replica has last-logged OpId of term: 6 index: 132, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 132. I1017 23:32:55.803594 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 7. I1017 23:32:55.804106 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:55.804175 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. I1017 23:32:55.804388 20283 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.205s (election complete) I1017 23:32:55.804507 20283 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 FOLLOWER]: Leader election won for term 7 I1017 23:32:55.804754 20283 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:55.805032 20283 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 5.132, Last appended by leader: 132, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:55.807274 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 6 to 7, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 7 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } I1017 23:32:55.807803 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40032 I1017 23:32:55.807919 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Received request to transfer leadership I1017 23:32:55.807970 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader W1017 23:32:56.080348 20293 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:56.138958 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35292 I1017 23:32:56.139108 19712 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 LEADER]: Received request to step down I1017 23:32:56.139153 19712 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 LEADER]: Stepping down as leader of term 7 I1017 23:32:56.139185 19712 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:56.139384 19712 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 7.133, Last appended by leader: 133, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:56.139556 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:32:56.139598 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:56.139673 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 7 FOLLOWER]: Advancing to term 8 I1017 23:32:56.139729 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:32:57.412016 20314 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:57.466358 20330 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:57.466467 20330 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.752s (starting election) I1017 23:32:57.466567 20330 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:57.466979 20330 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:57.467180 20330 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:57.467865 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 8 candidate_status { last_received { term: 6 index: 132 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:32:57.468063 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 8 because replica has last-logged OpId of term: 7 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 132. I1017 23:32:57.468147 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 8 candidate_status { last_received { term: 6 index: 132 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:57.468327 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.956s (vote granted) I1017 23:32:57.468433 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 7. I1017 23:32:57.468591 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 8 because replica has last-logged OpId of term: 7 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 132. I1017 23:32:57.468879 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:57.468941 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. I1017 23:32:57.469135 20330 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.194s (election complete) I1017 23:32:57.469228 20330 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Leader pre-election won for term 8 I1017 23:32:57.469275 20330 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:57.469337 20330 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.066s (starting election) I1017 23:32:57.469395 20330 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 7 FOLLOWER]: Advancing to term 8 I1017 23:32:57.470830 20330 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:57.471148 20330 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:57.471318 20330 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:57.471940 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 8 candidate_status { last_received { term: 6 index: 132 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:32:57.472092 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 8 because replica has last-logged OpId of term: 7 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 132. I1017 23:32:57.472117 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 8 candidate_status { last_received { term: 6 index: 132 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:57.472250 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 7 FOLLOWER]: Advancing to term 8 I1017 23:32:57.472317 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.523s (vote granted) I1017 23:32:57.472545 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35295 I1017 23:32:57.472718 19712 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Received request to transfer leadership I1017 23:32:57.472784 19712 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:32:57.472827 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 8 because replica has last-logged OpId of term: 7 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 132. I1017 23:32:57.474196 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 8. I1017 23:32:57.474835 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:57.474896 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. I1017 23:32:57.475076 20330 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.598s (election complete) I1017 23:32:57.475139 20330 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 FOLLOWER]: Leader election won for term 8 I1017 23:32:57.475528 20330 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:57.475844 20330 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 6.132, Last appended by leader: 132, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } Illegal state: not currently leaderI1017 23:32:57.478286 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 7 to 8, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 8 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:32:57.746769 20339 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:57.804407 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40039 I1017 23:32:57.804567 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 LEADER]: Received request to step down I1017 23:32:57.804620 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 LEADER]: Stepping down as leader of term 8 I1017 23:32:57.804651 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:32:57.804846 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 8.133, Last appended by leader: 133, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:57.804987 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:57.805030 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:32:57.805104 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 8 FOLLOWER]: Advancing to term 9 I1017 23:32:57.805164 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) I1017 23:32:58.998406 20357 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:58.998507 20357 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.160s (starting election) I1017 23:32:58.998636 20357 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:58.999164 20357 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:32:58.999385 20357 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:59.000315 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" candidate_term: 9 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:32:59.000315 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" candidate_term: 9 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:32:59.000589 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 9 because replica has last-logged OpId of term: 8 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:59.000593 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 9 because replica has last-logged OpId of term: 7 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:59.001339 19769 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 9 because replica has last-logged OpId of term: 8 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:59.001505 19767 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 9 because replica has last-logged OpId of term: 7 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:32:59.001605 19767 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. I1017 23:32:59.001842 20357 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.832s (election complete) I1017 23:32:59.001942 20357 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 8 FOLLOWER]: Advancing to term 9 I1017 23:32:59.003578 20357 raft_consensus.cc:2594] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority W1017 23:32:59.076956 20356 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:59.136660 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40041 I1017 23:32:59.136811 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Received request to transfer leadership I1017 23:32:59.136873 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leaderI1017 23:32:59.140970 20374 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:32:59.141088 20374 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.958s (starting election) I1017 23:32:59.141276 20374 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:59.141808 20374 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:59.142031 20374 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:59.142722 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 9 candidate_status { last_received { term: 7 index: 133 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:32:59.142828 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 9 candidate_status { last_received { term: 7 index: 133 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:32:59.142940 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 9 because replica has last-logged OpId of term: 8 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 133. I1017 23:32:59.143002 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.557s (vote granted) I1017 23:32:59.143070 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 9. I1017 23:32:59.143645 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:59.143782 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. I1017 23:32:59.143893 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 9 because replica has last-logged OpId of term: 8 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 133. I1017 23:32:59.143971 20374 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.538s (election complete) I1017 23:32:59.144110 20374 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Leader pre-election won for term 9 I1017 23:32:59.144219 20374 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:32:59.144304 20374 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.006s (starting election) I1017 23:32:59.144382 20374 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 8 FOLLOWER]: Advancing to term 9 I1017 23:32:59.146667 20374 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:59.147115 20374 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:59.147330 20374 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:32:59.147933 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 9 candidate_status { last_received { term: 7 index: 133 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:32:59.147985 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 9 candidate_status { last_received { term: 7 index: 133 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:32:59.148097 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.657s (vote granted) I1017 23:32:59.148149 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 9 because replica has last-logged OpId of term: 8 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 133. I1017 23:32:59.148742 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 9 because replica has last-logged OpId of term: 8 index: 133, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 133. I1017 23:32:59.149724 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 9. I1017 23:32:59.150143 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:32:59.150194 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. I1017 23:32:59.150365 20374 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.111s (election complete) I1017 23:32:59.150441 20374 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 FOLLOWER]: Leader election won for term 9 I1017 23:32:59.150770 20374 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:59.151077 20374 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 7.133, Last appended by leader: 133, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:59.153496 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 8 to 9, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 9 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:32:59.415118 20382 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:32:59.473239 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35301 I1017 23:32:59.473387 19712 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 LEADER]: Received request to step down I1017 23:32:59.473429 19712 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 LEADER]: Stepping down as leader of term 9 I1017 23:32:59.473464 19712 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:32:59.473692 19712 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 9.134, Last appended by leader: 134, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:32:59.473839 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:32:59.473879 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:32:59.473955 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 9 FOLLOWER]: Advancing to term 10 I1017 23:32:59.474011 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:00.751852 20399 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:00.806469 20416 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:00.806557 20416 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.640s (starting election) I1017 23:33:00.806690 20416 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:00.807078 20416 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:00.807293 20416 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:00.808025 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 10 candidate_status { last_received { term: 8 index: 133 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:00.808292 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 10 because replica has last-logged OpId of term: 9 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 133. I1017 23:33:00.808293 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 10 candidate_status { last_received { term: 8 index: 133 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:00.808444 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.887s (vote granted) I1017 23:33:00.808540 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 9. I1017 23:33:00.808970 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 10 because replica has last-logged OpId of term: 9 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 133. I1017 23:33:00.809150 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:00.809204 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. I1017 23:33:00.809397 20416 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.828s (election complete) I1017 23:33:00.809499 20416 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Leader pre-election won for term 10 I1017 23:33:00.809562 20416 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:00.809625 20416 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.229s (starting election) I1017 23:33:00.809685 20416 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 9 FOLLOWER]: Advancing to term 10 I1017 23:33:00.811111 20416 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:00.811523 20416 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:00.811702 20416 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:00.812511 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 10 candidate_status { last_received { term: 8 index: 133 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:00.812628 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 9 FOLLOWER]: Advancing to term 10 I1017 23:33:00.812616 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 10 candidate_status { last_received { term: 8 index: 133 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:00.812707 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.185s (vote granted) I1017 23:33:00.812894 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 10 because replica has last-logged OpId of term: 9 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 133. I1017 23:33:00.813408 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 10 because replica has last-logged OpId of term: 9 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 133. I1017 23:33:00.814028 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35303 I1017 23:33:00.814137 19712 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Received request to transfer leadership I1017 23:33:00.814186 19712 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:00.814471 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 10. I1017 23:33:00.814900 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:00.814956 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. I1017 23:33:00.815142 20416 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.505s (election complete) I1017 23:33:00.815217 20416 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 FOLLOWER]: Leader election won for term 10 I1017 23:33:00.816802 20416 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:00.817095 20416 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 8.133, Last appended by leader: 133, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:00.819397 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 9 to 10, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 10 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } Illegal state: not currently leader W1017 23:33:01.087764 20424 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:01.147269 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40047 I1017 23:33:01.147444 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 LEADER]: Received request to step down I1017 23:33:01.147502 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 LEADER]: Stepping down as leader of term 10 I1017 23:33:01.147550 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:01.147763 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 10.134, Last appended by leader: 134, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:01.147919 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:01.147979 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:01.148066 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 10 FOLLOWER]: Advancing to term 11 I1017 23:33:01.148138 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:02.425642 20451 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:02.474853 20462 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:02.474936 20462 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.549s (starting election) I1017 23:33:02.475029 20462 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:02.475450 20462 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:02.475687 20462 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:02.476457 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 11 candidate_status { last_received { term: 9 index: 134 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:02.476603 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 11 candidate_status { last_received { term: 9 index: 134 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:02.476680 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.070s (vote granted) I1017 23:33:02.476758 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 10. I1017 23:33:02.476832 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 11 because replica has last-logged OpId of term: 10 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 134. I1017 23:33:02.477411 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:02.477509 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. I1017 23:33:02.477674 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 11 because replica has last-logged OpId of term: 10 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 134. I1017 23:33:02.477955 20462 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.659s (election complete) I1017 23:33:02.478113 20462 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Leader pre-election won for term 11 I1017 23:33:02.478185 20462 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:02.478260 20462 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.646s (starting election) I1017 23:33:02.478341 20462 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 10 FOLLOWER]: Advancing to term 11 I1017 23:33:02.480455 20462 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:02.480907 20462 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:02.481094 20462 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:02.481734 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 11 candidate_status { last_received { term: 9 index: 134 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:02.481854 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 10 FOLLOWER]: Advancing to term 11 I1017 23:33:02.481925 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.960s (vote granted) I1017 23:33:02.481916 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 11 candidate_status { last_received { term: 9 index: 134 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:02.482139 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 11 because replica has last-logged OpId of term: 10 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 134. I1017 23:33:02.483914 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 11 because replica has last-logged OpId of term: 10 index: 134, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 134. I1017 23:33:02.484369 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 11. I1017 23:33:02.484773 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:02.484833 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. I1017 23:33:02.485019 20462 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.860s (election complete) I1017 23:33:02.485090 20462 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 FOLLOWER]: Leader election won for term 11 I1017 23:33:02.485383 20462 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:02.485688 20462 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 9.134, Last appended by leader: 134, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:02.487879 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 10 to 11, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 11 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } I1017 23:33:02.488633 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40052 I1017 23:33:02.488750 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Received request to transfer leadership I1017 23:33:02.488788 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader W1017 23:33:02.761781 20476 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:02.821249 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35312 I1017 23:33:02.821404 19712 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 LEADER]: Received request to step down I1017 23:33:02.821460 19712 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 LEADER]: Stepping down as leader of term 11 I1017 23:33:02.821532 19712 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:02.821774 19712 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 11.135, Last appended by leader: 135, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:02.821926 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:02.821983 19712 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:02.822079 19712 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 11 FOLLOWER]: Advancing to term 12 I1017 23:33:02.822144 19712 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:04.094280 20493 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:04.149832 20510 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:04.149966 20510 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.242s (starting election) I1017 23:33:04.150080 20510 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:04.150658 20510 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:04.150909 20510 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:04.151613 19712 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 12 candidate_status { last_received { term: 10 index: 134 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:04.151834 19712 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 12 because replica has last-logged OpId of term: 11 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 134. I1017 23:33:04.151856 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 12 candidate_status { last_received { term: 10 index: 134 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:04.152071 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.686s (vote granted) I1017 23:33:04.152168 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 11. I1017 23:33:04.152483 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 12 because replica has last-logged OpId of term: 11 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 134. I1017 23:33:04.152707 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:04.152763 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. I1017 23:33:04.152973 20510 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.232s (election complete) I1017 23:33:04.153122 20510 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Leader pre-election won for term 12 I1017 23:33:04.153201 20510 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:04.154108 20510 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.539s (starting election) I1017 23:33:04.154204 20510 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 11 FOLLOWER]: Advancing to term 12 I1017 23:33:04.156198 20510 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:04.156584 20510 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:04.156798 20510 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:04.157479 19712 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35314 I1017 23:33:04.157531 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 12 candidate_status { last_received { term: 10 index: 134 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:04.157644 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 12 candidate_status { last_received { term: 10 index: 134 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:04.157696 19712 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Received request to transfer leadership I1017 23:33:04.157742 19712 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:04.157757 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 11 FOLLOWER]: Advancing to term 12 I1017 23:33:04.157840 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.353s (vote granted) I1017 23:33:04.158073 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 12 because replica has last-logged OpId of term: 11 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 134. I1017 23:33:04.158509 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 12 because replica has last-logged OpId of term: 11 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 134. I1017 23:33:04.159809 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 12. I1017 23:33:04.160267 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:04.160333 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. I1017 23:33:04.160516 20510 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.912s (election complete) I1017 23:33:04.160588 20510 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 FOLLOWER]: Leader election won for term 12 I1017 23:33:04.161012 20510 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:04.161288 20510 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 10.134, Last appended by leader: 134, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } Illegal state: not currently leader I1017 23:33:04.163450 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 11 to 12, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 12 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:04.431114 20518 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:04.490387 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40058 I1017 23:33:04.490547 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 LEADER]: Received request to step down I1017 23:33:04.490593 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 LEADER]: Stepping down as leader of term 12 I1017 23:33:04.490618 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:04.490844 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 12.135, Last appended by leader: 135, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:04.490995 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:04.491053 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:04.491148 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 12 FOLLOWER]: Advancing to term 13 I1017 23:33:04.491225 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:05.765731 20535 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:05.823542 20552 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:05.823623 20552 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.766s (starting election) I1017 23:33:05.823731 20552 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:05.824132 20552 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:05.824332 20552 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:05.825031 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 13 candidate_status { last_received { term: 11 index: 135 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:05.825178 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.138s (vote granted) I1017 23:33:05.825253 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 12. I1017 23:33:05.825183 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 13 candidate_status { last_received { term: 11 index: 135 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:05.825547 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 13 because replica has last-logged OpId of term: 12 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 135. I1017 23:33:05.825769 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:05.825829 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. I1017 23:33:05.826032 20552 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.657s (election complete) I1017 23:33:05.826088 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 13 because replica has last-logged OpId of term: 12 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 135. I1017 23:33:05.826133 20552 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Leader pre-election won for term 13 I1017 23:33:05.826220 20552 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:05.826273 20552 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.577s (starting election) I1017 23:33:05.826328 20552 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 12 FOLLOWER]: Advancing to term 13 I1017 23:33:05.827286 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40060 I1017 23:33:05.827392 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Received request to transfer leadership I1017 23:33:05.827435 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:05.828367 20552 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:05.828835 20552 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:05.829068 20552 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:05.829605 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 13 candidate_status { last_received { term: 11 index: 135 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:05.829725 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 12 FOLLOWER]: Advancing to term 13 I1017 23:33:05.829790 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.697s (vote granted) I1017 23:33:05.829764 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 13 candidate_status { last_received { term: 11 index: 135 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:05.829897 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 13 because replica has last-logged OpId of term: 12 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 135. I1017 23:33:05.830417 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 13 because replica has last-logged OpId of term: 12 index: 135, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 135. I1017 23:33:05.831853 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 13. Illegal state: not currently leaderI1017 23:33:05.832304 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:05.832365 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. I1017 23:33:05.832619 20552 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.022s (election complete) I1017 23:33:05.832711 20552 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 FOLLOWER]: Leader election won for term 13 I1017 23:33:05.833000 20552 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:05.833283 20552 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 11.135, Last appended by leader: 135, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:05.835618 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 12 to 13, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 13 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:06.099984 20560 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:06.158658 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35320 I1017 23:33:06.158812 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 LEADER]: Received request to step down I1017 23:33:06.158876 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 LEADER]: Stepping down as leader of term 13 I1017 23:33:06.158921 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:06.159157 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 13.136, Last appended by leader: 136, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:06.159317 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:06.159370 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:06.159466 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 13 FOLLOWER]: Advancing to term 14 I1017 23:33:06.159533 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:07.433795 20581 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:07.492806 20598 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:07.492888 20598 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.609s (starting election) I1017 23:33:07.492997 20598 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:07.493398 20598 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:07.493618 20598 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:07.494300 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 14 candidate_status { last_received { term: 12 index: 135 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:07.494565 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 14 because replica has last-logged OpId of term: 13 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 135. I1017 23:33:07.494590 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 14 candidate_status { last_received { term: 12 index: 135 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:07.494765 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.600s (vote granted) I1017 23:33:07.494837 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 13. I1017 23:33:07.495085 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 14 because replica has last-logged OpId of term: 13 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 135. I1017 23:33:07.495319 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:07.495383 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. I1017 23:33:07.495494 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35323 I1017 23:33:07.495620 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Received request to transfer leadership I1017 23:33:07.495620 20598 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.213s (election complete) I1017 23:33:07.495654 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:07.495715 20598 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Leader pre-election won for term 14 I1017 23:33:07.495751 20598 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:07.495785 20598 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.936s (starting election) I1017 23:33:07.495818 20598 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 13 FOLLOWER]: Advancing to term 14 I1017 23:33:07.497793 20598 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:07.498159 20598 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) Illegal state: not currently leader I1017 23:33:07.498371 20598 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:07.499111 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 14 candidate_status { last_received { term: 12 index: 135 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:07.499081 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 14 candidate_status { last_received { term: 12 index: 135 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:07.499222 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 13 FOLLOWER]: Advancing to term 14 I1017 23:33:07.499285 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 14 because replica has last-logged OpId of term: 13 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 135. I1017 23:33:07.499296 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.130s (vote granted) I1017 23:33:07.499796 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 14 because replica has last-logged OpId of term: 13 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 135. I1017 23:33:07.501029 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 14. I1017 23:33:07.501497 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:07.501561 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. I1017 23:33:07.501760 20598 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.539s (election complete) I1017 23:33:07.501839 20598 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 FOLLOWER]: Leader election won for term 14 I1017 23:33:07.502151 20598 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:07.502710 20598 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 12.135, Last appended by leader: 135, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:07.505254 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 13 to 14, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 14 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:07.768005 20606 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:07.827873 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40067 I1017 23:33:07.828037 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 LEADER]: Received request to step down I1017 23:33:07.828094 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 LEADER]: Stepping down as leader of term 14 I1017 23:33:07.828126 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:07.828346 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 14.136, Last appended by leader: 136, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:07.828517 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:07.828582 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:07.828662 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 14 FOLLOWER]: Advancing to term 15 I1017 23:33:07.828725 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:09.100044 20623 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:09.159790 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40069 I1017 23:33:09.159924 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Received request to transfer leadership I1017 23:33:09.159974 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:09.160842 20640 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:09.160904 20640 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.015s (starting election) I1017 23:33:09.160992 20640 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:09.161535 20640 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:09.161727 20640 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:09.162436 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 15 candidate_status { last_received { term: 13 index: 136 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:09.162611 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.052s (vote granted) I1017 23:33:09.162685 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 14. I1017 23:33:09.163007 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 15 candidate_status { last_received { term: 13 index: 136 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:09.163184 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:09.163202 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 15 because replica has last-logged OpId of term: 14 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 136. I1017 23:33:09.163241 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. I1017 23:33:09.163792 20640 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.034s (election complete) I1017 23:33:09.163792 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 15 because replica has last-logged OpId of term: 14 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 136. I1017 23:33:09.163930 20640 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Leader pre-election won for term 15 I1017 23:33:09.164121 20640 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:09.164180 20640 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.165s (starting election) I1017 23:33:09.164239 20640 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 14 FOLLOWER]: Advancing to term 15 I1017 23:33:09.166100 20640 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:09.166404 20640 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:09.166661 20640 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) Illegal state: not currently leader I1017 23:33:09.167325 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 15 candidate_status { last_received { term: 13 index: 136 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:09.167598 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 14 FOLLOWER]: Advancing to term 15 I1017 23:33:09.167703 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.773s (vote granted) I1017 23:33:09.167896 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 15 candidate_status { last_received { term: 13 index: 136 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:09.168077 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 15 because replica has last-logged OpId of term: 14 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 136. I1017 23:33:09.168659 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 15 because replica has last-logged OpId of term: 14 index: 136, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 136. I1017 23:33:09.169770 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 15. I1017 23:33:09.170161 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:09.170219 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. I1017 23:33:09.170413 20640 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.856s (election complete) I1017 23:33:09.170490 20640 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 FOLLOWER]: Leader election won for term 15 I1017 23:33:09.170835 20640 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:09.171114 20640 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 13.136, Last appended by leader: 136, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:09.173606 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 14 to 15, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 15 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:09.436681 20648 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:09.496572 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35329 I1017 23:33:09.496738 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 LEADER]: Received request to step down I1017 23:33:09.496793 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 LEADER]: Stepping down as leader of term 15 I1017 23:33:09.496825 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:09.497084 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 15.137, Last appended by leader: 137, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:09.497288 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:09.497344 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:09.497437 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 15 FOLLOWER]: Advancing to term 16 I1017 23:33:09.497527 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:10.767992 20665 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:10.828366 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35331 I1017 23:33:10.828541 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Received request to transfer leadership I1017 23:33:10.828604 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:10.829375 20682 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:10.829483 20682 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.152s (starting election) I1017 23:33:10.829588 20682 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:10.830212 20682 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:10.830426 20682 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:10.831005 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 16 candidate_status { last_received { term: 14 index: 136 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:10.831224 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 16 because replica has last-logged OpId of term: 15 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 136. I1017 23:33:10.831297 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 16 candidate_status { last_received { term: 14 index: 136 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:10.831495 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.933s (vote granted) I1017 23:33:10.831598 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 15. I1017 23:33:10.832260 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:10.832324 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. I1017 23:33:10.832644 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 16 because replica has last-logged OpId of term: 15 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 136. I1017 23:33:10.832811 20682 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.829s (election complete) I1017 23:33:10.832911 20682 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Leader pre-election won for term 16 I1017 23:33:10.832952 20682 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:10.832988 20682 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.133s (starting election) I1017 23:33:10.833024 20682 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 15 FOLLOWER]: Advancing to term 16 I1017 23:33:10.834394 20682 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:10.834704 20682 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:10.834827 20682 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) Illegal state: not currently leader I1017 23:33:10.835533 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 16 candidate_status { last_received { term: 14 index: 136 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:10.835705 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 16 because replica has last-logged OpId of term: 15 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 136. I1017 23:33:10.835695 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 16 candidate_status { last_received { term: 14 index: 136 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:10.835781 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 15 FOLLOWER]: Advancing to term 16 I1017 23:33:10.835849 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.257s (vote granted) I1017 23:33:10.836251 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 16 because replica has last-logged OpId of term: 15 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 136. I1017 23:33:10.837386 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 16. I1017 23:33:10.837805 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:10.837862 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. I1017 23:33:10.838062 20682 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.906s (election complete) I1017 23:33:10.838135 20682 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 FOLLOWER]: Leader election won for term 16 I1017 23:33:10.838479 20682 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:10.838943 20682 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 14.136, Last appended by leader: 136, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:10.841156 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 15 to 16, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 16 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:11.101492 20690 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:11.160044 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40075 I1017 23:33:11.160181 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 LEADER]: Received request to step down I1017 23:33:11.160230 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 LEADER]: Stepping down as leader of term 16 I1017 23:33:11.160260 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:11.160504 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 16.137, Last appended by leader: 137, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:11.160652 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:11.160701 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:11.160776 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 16 FOLLOWER]: Advancing to term 17 I1017 23:33:11.160841 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:12.434813 20711 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:12.493705 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40078 I1017 23:33:12.493854 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Received request to transfer leadership I1017 23:33:12.493907 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader I1017 23:33:12.499171 20728 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:12.499253 20728 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.547s (starting election) I1017 23:33:12.499347 20728 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:12.499792 20728 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:12.499979 20728 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:12.500715 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 17 candidate_status { last_received { term: 15 index: 137 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:12.500891 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 17 because replica has last-logged OpId of term: 16 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 137. I1017 23:33:12.500830 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 17 candidate_status { last_received { term: 15 index: 137 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:12.501000 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.208s (vote granted) I1017 23:33:12.501075 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 16. I1017 23:33:12.501410 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 17 because replica has last-logged OpId of term: 16 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 137. I1017 23:33:12.501767 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:12.501839 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. I1017 23:33:12.502038 20728 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.674s (election complete) I1017 23:33:12.502142 20728 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Leader pre-election won for term 17 I1017 23:33:12.502192 20728 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:12.502249 20728 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.237s (starting election) I1017 23:33:12.502313 20728 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 16 FOLLOWER]: Advancing to term 17 I1017 23:33:12.503780 20728 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:12.504093 20728 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:12.504276 20728 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:12.504892 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 17 candidate_status { last_received { term: 15 index: 137 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:12.504938 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 17 candidate_status { last_received { term: 15 index: 137 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:12.505002 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 16 FOLLOWER]: Advancing to term 17 I1017 23:33:12.505061 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.759s (vote granted) I1017 23:33:12.505133 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 17 because replica has last-logged OpId of term: 16 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 137. I1017 23:33:12.505576 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 17 because replica has last-logged OpId of term: 16 index: 137, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 137. I1017 23:33:12.506399 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 17. I1017 23:33:12.506820 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:12.506882 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. I1017 23:33:12.507089 20728 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.626s (election complete) I1017 23:33:12.507161 20728 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 FOLLOWER]: Leader election won for term 17 I1017 23:33:12.507549 20728 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:12.507992 20728 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 15.137, Last appended by leader: 137, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:12.510308 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 16 to 17, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 17 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:12.771525 20736 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:12.830624 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35338 I1017 23:33:12.830780 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 LEADER]: Received request to step down I1017 23:33:12.830842 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 LEADER]: Stepping down as leader of term 17 I1017 23:33:12.830885 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:12.831133 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 17.138, Last appended by leader: 138, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:12.831285 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:12.831337 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:12.831449 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 17 FOLLOWER]: Advancing to term 18 I1017 23:33:12.831519 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:14.105149 20753 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:14.162416 20770 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:14.162533 20770 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.678s (starting election) I1017 23:33:14.162629 20770 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:14.163127 20770 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:14.163360 20770 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:14.164065 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 18 candidate_status { last_received { term: 16 index: 137 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:14.164266 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 18 because replica has last-logged OpId of term: 17 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 137. I1017 23:33:14.164300 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 18 candidate_status { last_received { term: 16 index: 137 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:14.164443 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.248s (vote granted) I1017 23:33:14.164535 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 17. I1017 23:33:14.164888 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 18 because replica has last-logged OpId of term: 17 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 137. I1017 23:33:14.165136 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:14.165205 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. I1017 23:33:14.165863 20770 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.985s (election complete) I1017 23:33:14.165943 20770 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Leader pre-election won for term 18 I1017 23:33:14.165999 20770 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:14.166050 20770 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.935s (starting election) I1017 23:33:14.166110 20770 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 17 FOLLOWER]: Advancing to term 18 I1017 23:33:14.168049 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35340 I1017 23:33:14.168186 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Received request to transfer leadership I1017 23:33:14.168242 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:14.168267 20770 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:14.168658 20770 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:14.169023 20770 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:14.169333 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 18 candidate_status { last_received { term: 16 index: 137 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:14.169548 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 18 because replica has last-logged OpId of term: 17 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 137. I1017 23:33:14.169771 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 18 candidate_status { last_received { term: 16 index: 137 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:14.169901 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 17 FOLLOWER]: Advancing to term 18 I1017 23:33:14.170020 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.350s (vote granted) I1017 23:33:14.171571 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 18 because replica has last-logged OpId of term: 17 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 137. Illegal state: not currently leader I1017 23:33:14.172941 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 18. I1017 23:33:14.173357 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:14.173408 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. I1017 23:33:14.173616 20770 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.669s (election complete) I1017 23:33:14.173693 20770 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 FOLLOWER]: Leader election won for term 18 I1017 23:33:14.174026 20770 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:14.174343 20770 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 16.137, Last appended by leader: 137, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:14.176796 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 17 to 18, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 18 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:14.442530 20778 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:14.501570 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40084 I1017 23:33:14.501724 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 LEADER]: Received request to step down I1017 23:33:14.501791 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 LEADER]: Stepping down as leader of term 18 I1017 23:33:14.501837 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:14.502076 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 18.138, Last appended by leader: 138, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:14.502239 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:14.502295 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:14.502367 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 18 FOLLOWER]: Advancing to term 19 I1017 23:33:14.502423 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:15.775527 20795 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:15.832731 20812 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:15.832809 20812 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.770s (starting election) I1017 23:33:15.832913 20812 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:15.833323 20812 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:15.833514 20812 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:15.834283 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 19 candidate_status { last_received { term: 17 index: 138 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:15.834436 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 19 candidate_status { last_received { term: 17 index: 138 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:15.834589 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 19 because replica has last-logged OpId of term: 18 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 138. I1017 23:33:15.834594 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.184s (vote granted) I1017 23:33:15.834668 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 18. I1017 23:33:15.835270 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 19 because replica has last-logged OpId of term: 18 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 138. I1017 23:33:15.835469 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:15.835547 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. I1017 23:33:15.835757 20812 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.782s (election complete) I1017 23:33:15.835855 20812 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Leader pre-election won for term 19 I1017 23:33:15.835903 20812 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:15.835960 20812 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.548s (starting election) I1017 23:33:15.836014 20812 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 18 FOLLOWER]: Advancing to term 19 I1017 23:33:15.836217 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40086 I1017 23:33:15.836342 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Received request to transfer leadership I1017 23:33:15.836390 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:15.838100 20812 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:15.838529 20812 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:15.838713 20812 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:15.839195 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 19 candidate_status { last_received { term: 17 index: 138 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:15.839310 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 18 FOLLOWER]: Advancing to term 19 I1017 23:33:15.839367 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.832s (vote granted) I1017 23:33:15.839354 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 19 candidate_status { last_received { term: 17 index: 138 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:15.839536 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 19 because replica has last-logged OpId of term: 18 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 138. I1017 23:33:15.839934 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 19 because replica has last-logged OpId of term: 18 index: 138, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 138. I1017 23:33:15.841066 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 19. I1017 23:33:15.841594 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:15.841658 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. I1017 23:33:15.841876 20812 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.204s (election complete) I1017 23:33:15.841951 20812 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 FOLLOWER]: Leader election won for term 19 I1017 23:33:15.842268 20812 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER Illegal state: not currently leader I1017 23:33:15.842563 20812 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 17.138, Last appended by leader: 138, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:15.845093 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 18 to 19, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 19 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:16.108085 20820 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:16.166610 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35346 I1017 23:33:16.166757 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 LEADER]: Received request to step down I1017 23:33:16.166822 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 LEADER]: Stepping down as leader of term 19 I1017 23:33:16.166869 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:16.167096 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 19.139, Last appended by leader: 139, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:16.167240 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:16.167300 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:16.167397 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 19 FOLLOWER]: Advancing to term 20 I1017 23:33:16.167486 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:17.443194 20841 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:17.503674 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35349 I1017 23:33:17.503721 20858 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:17.503805 20858 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.859s (starting election) I1017 23:33:17.503850 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Received request to transfer leadership I1017 23:33:17.503919 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:17.503906 20858 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:17.504354 20858 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:17.504570 20858 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:17.505249 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 20 candidate_status { last_received { term: 18 index: 138 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:17.505491 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 20 because replica has last-logged OpId of term: 19 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 138. I1017 23:33:17.505631 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 20 candidate_status { last_received { term: 18 index: 138 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:17.505816 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.099s (vote granted) I1017 23:33:17.505887 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 19. I1017 23:33:17.506034 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 20 because replica has last-logged OpId of term: 19 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 138. I1017 23:33:17.506387 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:17.506469 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. I1017 23:33:17.506677 20858 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.839s (election complete) I1017 23:33:17.506748 20858 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Leader pre-election won for term 20 I1017 23:33:17.506800 20858 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:17.506876 20858 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.222s (starting election) I1017 23:33:17.506930 20858 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 19 FOLLOWER]: Advancing to term 20 Illegal state: not currently leaderI1017 23:33:17.508786 20858 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:17.509133 20858 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:17.509310 20858 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:17.509935 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 20 candidate_status { last_received { term: 18 index: 138 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:17.510119 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 20 because replica has last-logged OpId of term: 19 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 138. I1017 23:33:17.510087 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 20 candidate_status { last_received { term: 18 index: 138 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:17.510211 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 19 FOLLOWER]: Advancing to term 20 I1017 23:33:17.510293 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.825s (vote granted) I1017 23:33:17.510653 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 20 because replica has last-logged OpId of term: 19 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 138. I1017 23:33:17.511757 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 20. I1017 23:33:17.512143 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:17.512200 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. I1017 23:33:17.512390 20858 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.177s (election complete) I1017 23:33:17.512488 20858 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 FOLLOWER]: Leader election won for term 20 I1017 23:33:17.512799 20858 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:17.513059 20858 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 18.138, Last appended by leader: 138, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:17.515266 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 19 to 20, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 20 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:17.777114 20866 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:17.835691 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40093 I1017 23:33:17.835853 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 LEADER]: Received request to step down I1017 23:33:17.835916 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 LEADER]: Stepping down as leader of term 20 I1017 23:33:17.835963 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:17.836210 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 20.139, Last appended by leader: 139, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:17.836359 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:17.836417 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:17.836529 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 20 FOLLOWER]: Advancing to term 21 I1017 23:33:17.836606 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:19.108014 20883 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:19.167443 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40095 I1017 23:33:19.167609 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Received request to transfer leadership I1017 23:33:19.167666 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:19.168382 20900 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:19.168454 20900 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.014s (starting election) I1017 23:33:19.168565 20900 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:19.168958 20900 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:19.169270 20900 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:19.169939 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 21 candidate_status { last_received { term: 19 index: 139 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:19.169986 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 21 candidate_status { last_received { term: 19 index: 139 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:19.170132 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.224s (vote granted) I1017 23:33:19.170163 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 21 because replica has last-logged OpId of term: 20 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 139. I1017 23:33:19.170212 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 20. I1017 23:33:19.170660 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 21 because replica has last-logged OpId of term: 20 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 139. I1017 23:33:19.170778 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:19.170840 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. I1017 23:33:19.171051 20900 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.925s (election complete) I1017 23:33:19.171118 20900 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Leader pre-election won for term 21 I1017 23:33:19.171186 20900 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:19.171244 20900 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.820s (starting election) I1017 23:33:19.171300 20900 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 20 FOLLOWER]: Advancing to term 21 I1017 23:33:19.172960 20900 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:19.173316 20900 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:19.173503 20900 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) Illegal state: not currently leader I1017 23:33:19.173969 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 21 candidate_status { last_received { term: 19 index: 139 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:19.174223 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 20 FOLLOWER]: Advancing to term 21 I1017 23:33:19.174201 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 21 candidate_status { last_received { term: 19 index: 139 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:19.174327 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.747s (vote granted) I1017 23:33:19.174379 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 21 because replica has last-logged OpId of term: 20 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 139. I1017 23:33:19.174787 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 21 because replica has last-logged OpId of term: 20 index: 139, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 139. I1017 23:33:19.176355 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 21. I1017 23:33:19.176837 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:19.176895 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. I1017 23:33:19.177088 20900 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.921s (election complete) I1017 23:33:19.177145 20900 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 FOLLOWER]: Leader election won for term 21 I1017 23:33:19.177522 20900 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:19.177947 20900 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 19.139, Last appended by leader: 139, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:19.180160 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 20 to 21, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 21 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:19.441700 20908 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:19.501699 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35355 I1017 23:33:19.501844 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 LEADER]: Received request to step down I1017 23:33:19.501899 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 LEADER]: Stepping down as leader of term 21 I1017 23:33:19.501925 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:19.502115 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 21.140, Last appended by leader: 140, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:19.502259 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:19.502306 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:19.502373 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 21 FOLLOWER]: Advancing to term 22 I1017 23:33:19.502435 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:20.773846 20925 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:20.833204 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35357 I1017 23:33:20.833350 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Received request to transfer leadership I1017 23:33:20.833405 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leaderI1017 23:33:20.837913 20942 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:20.838086 20942 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.184s (starting election) I1017 23:33:20.838249 20942 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:20.838673 20942 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:20.838884 20942 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:20.839581 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 22 candidate_status { last_received { term: 20 index: 139 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:20.839808 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 22 because replica has last-logged OpId of term: 21 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 139. I1017 23:33:20.839787 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 22 candidate_status { last_received { term: 20 index: 139 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:20.839969 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.673s (vote granted) I1017 23:33:20.840041 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 21. I1017 23:33:20.840493 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 22 because replica has last-logged OpId of term: 21 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 139. I1017 23:33:20.840759 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:20.840829 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. I1017 23:33:20.841039 20942 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.227s (election complete) I1017 23:33:20.841136 20942 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Leader pre-election won for term 22 I1017 23:33:20.841184 20942 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:20.841240 20942 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.155s (starting election) I1017 23:33:20.841295 20942 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 21 FOLLOWER]: Advancing to term 22 I1017 23:33:20.842777 20942 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:20.843078 20942 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:20.843240 20942 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:20.843818 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 22 candidate_status { last_received { term: 20 index: 139 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:20.844017 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 22 because replica has last-logged OpId of term: 21 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 139. I1017 23:33:20.844044 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 22 candidate_status { last_received { term: 20 index: 139 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:20.844177 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 21 FOLLOWER]: Advancing to term 22 I1017 23:33:20.844251 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.171s (vote granted) I1017 23:33:20.844552 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 22 because replica has last-logged OpId of term: 21 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 139. I1017 23:33:20.845674 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 22. I1017 23:33:20.846127 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:20.846189 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. I1017 23:33:20.846374 20942 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.048s (election complete) I1017 23:33:20.846488 20942 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 FOLLOWER]: Leader election won for term 22 I1017 23:33:20.846909 20942 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:20.847208 20942 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 20.139, Last appended by leader: 139, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:20.849664 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 21 to 22, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 22 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:21.107239 20950 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:21.165248 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40101 I1017 23:33:21.165395 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 LEADER]: Received request to step down I1017 23:33:21.165468 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 LEADER]: Stepping down as leader of term 22 I1017 23:33:21.165501 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:21.165719 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 22.140, Last appended by leader: 140, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:21.165870 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:21.165915 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:21.165989 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 22 FOLLOWER]: Advancing to term 23 I1017 23:33:21.166044 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:22.439201 20971 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:22.498697 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40104 I1017 23:33:22.498843 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Received request to transfer leadership I1017 23:33:22.498904 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader I1017 23:33:22.503638 20988 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:22.503702 20988 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.904s (starting election) I1017 23:33:22.503772 20988 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:22.504159 20988 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:22.504325 20988 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:22.504981 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 23 candidate_status { last_received { term: 21 index: 140 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:22.505146 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 23 because replica has last-logged OpId of term: 22 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 140. I1017 23:33:22.505164 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 23 candidate_status { last_received { term: 21 index: 140 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:22.505342 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.599s (vote granted) I1017 23:33:22.505416 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 22. I1017 23:33:22.505650 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 23 because replica has last-logged OpId of term: 22 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 140. I1017 23:33:22.506069 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:22.506131 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. I1017 23:33:22.506330 20988 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.502s (election complete) I1017 23:33:22.506414 20988 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Leader pre-election won for term 23 I1017 23:33:22.506460 20988 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:22.506518 20988 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.056s (starting election) I1017 23:33:22.506556 20988 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 22 FOLLOWER]: Advancing to term 23 I1017 23:33:22.508070 20988 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:22.508468 20988 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:22.508677 20988 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:22.509192 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 23 candidate_status { last_received { term: 21 index: 140 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:22.509310 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 22 FOLLOWER]: Advancing to term 23 I1017 23:33:22.509299 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 23 candidate_status { last_received { term: 21 index: 140 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:22.509377 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.905s (vote granted) I1017 23:33:22.509457 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 23 because replica has last-logged OpId of term: 22 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 140. I1017 23:33:22.509884 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 23 because replica has last-logged OpId of term: 22 index: 140, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 140. I1017 23:33:22.510782 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 23. I1017 23:33:22.511163 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:22.511221 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. I1017 23:33:22.511409 20988 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.921s (election complete) I1017 23:33:22.511508 20988 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 FOLLOWER]: Leader election won for term 23 I1017 23:33:22.511822 20988 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:22.512104 20988 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 21.140, Last appended by leader: 140, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:22.514708 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 22 to 23, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 23 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:22.774174 20996 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:22.832926 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35364 I1017 23:33:22.833086 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 LEADER]: Received request to step down I1017 23:33:22.833148 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 LEADER]: Stepping down as leader of term 23 I1017 23:33:22.833190 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:22.833425 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 23.141, Last appended by leader: 141, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:22.833590 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:22.833642 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:22.833709 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 23 FOLLOWER]: Advancing to term 24 I1017 23:33:22.833763 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:24.104557 21013 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:24.163753 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35366 I1017 23:33:24.163905 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Received request to transfer leadership I1017 23:33:24.163964 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:24.167490 21030 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:24.167608 21030 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.030s (starting election) I1017 23:33:24.167697 21030 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:24.168278 21030 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:24.168581 21030 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:24.169196 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 24 candidate_status { last_received { term: 22 index: 140 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:24.169416 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 24 because replica has last-logged OpId of term: 23 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 140. I1017 23:33:24.169497 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 24 candidate_status { last_received { term: 22 index: 140 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:24.169642 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.062s (vote granted) I1017 23:33:24.169709 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 23. I1017 23:33:24.169988 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 24 because replica has last-logged OpId of term: 23 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 140. I1017 23:33:24.170274 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:24.170337 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. I1017 23:33:24.170574 21030 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.982s (election complete) Illegal state: not currently leaderI1017 23:33:24.170650 21030 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Leader pre-election won for term 24 I1017 23:33:24.170696 21030 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:24.171193 21030 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.876s (starting election) I1017 23:33:24.171274 21030 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 23 FOLLOWER]: Advancing to term 24 I1017 23:33:24.172706 21030 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:24.173023 21030 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:24.173193 21030 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:24.173779 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 24 candidate_status { last_received { term: 22 index: 140 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:24.173933 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 24 candidate_status { last_received { term: 22 index: 140 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:24.173977 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 24 because replica has last-logged OpId of term: 23 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 140. I1017 23:33:24.174041 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 23 FOLLOWER]: Advancing to term 24 I1017 23:33:24.174113 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.316s (vote granted) I1017 23:33:24.174517 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 24 because replica has last-logged OpId of term: 23 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 140. I1017 23:33:24.175635 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 24. I1017 23:33:24.176106 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:24.176162 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. I1017 23:33:24.176353 21030 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.626s (election complete) I1017 23:33:24.176448 21030 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 FOLLOWER]: Leader election won for term 24 I1017 23:33:24.176800 21030 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:24.177086 21030 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 22.140, Last appended by leader: 140, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:24.179527 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 23 to 24, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 24 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:24.436842 21038 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:24.499083 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40110 I1017 23:33:24.499239 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 LEADER]: Received request to step down I1017 23:33:24.499300 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 LEADER]: Stepping down as leader of term 24 I1017 23:33:24.499341 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:24.499593 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 24.141, Last appended by leader: 141, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:24.499770 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:24.499824 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:24.499917 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 24 FOLLOWER]: Advancing to term 25 I1017 23:33:24.499990 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:25.773572 21055 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:25.831809 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40112 I1017 23:33:25.831964 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Received request to transfer leadership I1017 23:33:25.832031 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:25.834292 21072 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:25.834375 21072 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.163s (starting election) I1017 23:33:25.834478 21072 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:25.835144 21072 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:25.835348 21072 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:25.836081 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 25 candidate_status { last_received { term: 23 index: 141 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:25.836246 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.001s (vote granted) I1017 23:33:25.836258 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 25 candidate_status { last_received { term: 23 index: 141 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:25.836319 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 24. I1017 23:33:25.836402 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 25 because replica has last-logged OpId of term: 24 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 141. Illegal state: not currently leaderI1017 23:33:25.836905 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:25.836994 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. I1017 23:33:25.837090 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 25 because replica has last-logged OpId of term: 24 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 141. I1017 23:33:25.837203 21072 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.668s (election complete) I1017 23:33:25.837263 21072 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Leader pre-election won for term 25 I1017 23:33:25.837301 21072 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:25.837342 21072 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.574s (starting election) I1017 23:33:25.837389 21072 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 24 FOLLOWER]: Advancing to term 25 I1017 23:33:25.838824 21072 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:25.839124 21072 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:25.839273 21072 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:25.839851 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 25 candidate_status { last_received { term: 23 index: 141 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:25.839936 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 25 candidate_status { last_received { term: 23 index: 141 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:25.840000 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 24 FOLLOWER]: Advancing to term 25 I1017 23:33:25.840062 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.440s (vote granted) I1017 23:33:25.840083 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 25 because replica has last-logged OpId of term: 24 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 141. I1017 23:33:25.840557 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 25 because replica has last-logged OpId of term: 24 index: 141, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 141. I1017 23:33:25.841596 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 25. I1017 23:33:25.842097 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:25.842151 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. I1017 23:33:25.842342 21072 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.534s (election complete) I1017 23:33:25.842856 21072 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 FOLLOWER]: Leader election won for term 25 I1017 23:33:25.843137 21072 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:25.843433 21072 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 23.141, Last appended by leader: 141, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:25.845794 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 24 to 25, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 25 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:26.104940 21080 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:26.163869 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35372 I1017 23:33:26.164032 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 LEADER]: Received request to step down I1017 23:33:26.164098 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 LEADER]: Stepping down as leader of term 25 I1017 23:33:26.164139 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:26.164381 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 25.142, Last appended by leader: 142, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:26.164543 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:26.164600 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:26.164675 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 25 FOLLOWER]: Advancing to term 26 I1017 23:33:26.164744 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:27.436125 21101 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:27.496881 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35375 I1017 23:33:27.497046 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Received request to transfer leadership I1017 23:33:27.497112 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leaderI1017 23:33:27.500583 21118 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:27.500684 21118 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.947s (starting election) I1017 23:33:27.500768 21118 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:27.501206 21118 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:27.501399 21118 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:27.502007 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 26 candidate_status { last_received { term: 24 index: 141 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:27.502180 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 26 because replica has last-logged OpId of term: 25 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 141. I1017 23:33:27.502497 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 26 candidate_status { last_received { term: 24 index: 141 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:27.502691 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.225s (vote granted) I1017 23:33:27.502749 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 25. I1017 23:33:27.502882 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 26 because replica has last-logged OpId of term: 25 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 141. I1017 23:33:27.503232 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:27.503304 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. I1017 23:33:27.503526 21118 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.668s (election complete) I1017 23:33:27.503623 21118 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Leader pre-election won for term 26 I1017 23:33:27.503670 21118 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:27.503768 21118 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.157s (starting election) I1017 23:33:27.503821 21118 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 25 FOLLOWER]: Advancing to term 26 I1017 23:33:27.505246 21118 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:27.505594 21118 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:27.505754 21118 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:27.506335 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 26 candidate_status { last_received { term: 24 index: 141 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:27.506546 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 26 because replica has last-logged OpId of term: 25 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 141. I1017 23:33:27.506505 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 26 candidate_status { last_received { term: 24 index: 141 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:27.506613 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 25 FOLLOWER]: Advancing to term 26 I1017 23:33:27.506669 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.815s (vote granted) I1017 23:33:27.507097 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 26 because replica has last-logged OpId of term: 25 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 141. I1017 23:33:27.508124 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 26. I1017 23:33:27.508581 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:27.508637 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. I1017 23:33:27.508836 21118 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.234s (election complete) I1017 23:33:27.508911 21118 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 FOLLOWER]: Leader election won for term 26 I1017 23:33:27.509227 21118 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:27.509534 21118 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 24.141, Last appended by leader: 141, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:27.511703 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 25 to 26, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 26 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:27.768345 21126 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:27.827823 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40119 I1017 23:33:27.827985 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 LEADER]: Received request to step down I1017 23:33:27.828047 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 LEADER]: Stepping down as leader of term 26 I1017 23:33:27.828114 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:27.828361 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 26.142, Last appended by leader: 142, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:27.828528 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:27.828588 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:27.828673 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 26 FOLLOWER]: Advancing to term 27 I1017 23:33:27.828747 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:29.100369 21143 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:29.159366 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40121 I1017 23:33:29.159539 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Received request to transfer leadership I1017 23:33:29.159598 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader I1017 23:33:29.166210 21160 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:29.166278 21160 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.698s (starting election) I1017 23:33:29.166373 21160 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:29.166816 21160 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:29.167016 21160 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:29.167948 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 27 candidate_status { last_received { term: 25 index: 142 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:29.168159 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.057s (vote granted) I1017 23:33:29.168138 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 27 candidate_status { last_received { term: 25 index: 142 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:29.168272 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 26. I1017 23:33:29.168334 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 27 because replica has last-logged OpId of term: 26 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 142. I1017 23:33:29.169054 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 27 because replica has last-logged OpId of term: 26 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 142. I1017 23:33:29.169312 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:29.169378 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. I1017 23:33:29.169703 21160 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.608s (election complete) I1017 23:33:29.169778 21160 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Leader pre-election won for term 27 I1017 23:33:29.169826 21160 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:29.169875 21160 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.228s (starting election) I1017 23:33:29.169925 21160 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 26 FOLLOWER]: Advancing to term 27 I1017 23:33:29.172049 21160 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:29.172473 21160 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:29.172853 21160 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:29.173315 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 27 candidate_status { last_received { term: 25 index: 142 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:29.173524 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 26 FOLLOWER]: Advancing to term 27 I1017 23:33:29.173605 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.530s (vote granted) I1017 23:33:29.173561 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 27 candidate_status { last_received { term: 25 index: 142 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:29.173758 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 27 because replica has last-logged OpId of term: 26 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 142. I1017 23:33:29.174212 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 27 because replica has last-logged OpId of term: 26 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 142. I1017 23:33:29.175637 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 27. I1017 23:33:29.176051 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:29.176110 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. I1017 23:33:29.176327 21160 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.712s (election complete) I1017 23:33:29.176405 21160 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 FOLLOWER]: Leader election won for term 27 I1017 23:33:29.176746 21160 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:29.177001 21160 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 25.142, Last appended by leader: 142, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:29.179400 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 26 to 27, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 27 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:29.434721 21168 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:29.494961 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35381 I1017 23:33:29.495115 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 LEADER]: Received request to step down I1017 23:33:29.495174 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 LEADER]: Stepping down as leader of term 27 I1017 23:33:29.495223 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:29.495489 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 27.143, Last appended by leader: 143, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:29.495640 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:29.495699 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:29.495786 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 27 FOLLOWER]: Advancing to term 28 I1017 23:33:29.495862 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) I1017 23:33:30.706382 21186 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:30.706486 21186 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.317s (starting election) I1017 23:33:30.706637 21186 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:30.707114 21186 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:30.707319 21186 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:30.708335 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" candidate_term: 28 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:30.708559 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 28 because replica has last-logged OpId of term: 27 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:33:30.708446 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" candidate_term: 28 candidate_status { last_received { term: 2 index: 130 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:30.708737 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 28 because replica has last-logged OpId of term: 26 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:33:30.709100 19767 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 28 because replica has last-logged OpId of term: 27 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:33:30.709354 19769 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b01d528fd3c74eb5b42b8d4888591ed2 for term 28 because replica has last-logged OpId of term: 26 index: 142, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 130. I1017 23:33:30.709406 19769 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. I1017 23:33:30.709656 21186 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.650s (election complete) I1017 23:33:30.709727 21186 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 27 FOLLOWER]: Advancing to term 28 I1017 23:33:30.711158 21186 raft_consensus.cc:2594] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority W1017 23:33:30.772138 21185 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:30.829275 21203 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:30.829354 21203 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.506s (starting election) I1017 23:33:30.829458 21203 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:30.829869 21203 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:30.830070 21203 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:30.830853 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 28 candidate_status { last_received { term: 26 index: 142 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:30.831022 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 28 candidate_status { last_received { term: 26 index: 142 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:30.831148 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 28 because replica has last-logged OpId of term: 27 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 142. I1017 23:33:30.831204 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.410s (vote granted) I1017 23:33:30.831254 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 28. I1017 23:33:30.831703 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 28 because replica has last-logged OpId of term: 27 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 142. I1017 23:33:30.831926 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:30.831987 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. I1017 23:33:30.832201 21203 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.983s (election complete) I1017 23:33:30.832275 21203 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Leader pre-election won for term 28 I1017 23:33:30.832321 21203 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:30.832376 21203 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.689s (starting election) I1017 23:33:30.832427 21203 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 27 FOLLOWER]: Advancing to term 28 I1017 23:33:30.832871 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35383 I1017 23:33:30.832996 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Received request to transfer leadership I1017 23:33:30.833039 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:30.834298 21203 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:30.834738 21203 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:30.834894 21203 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:30.835695 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 28 candidate_status { last_received { term: 26 index: 142 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:30.835830 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 4.023s (vote granted) I1017 23:33:30.835896 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 28 candidate_status { last_received { term: 26 index: 142 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:30.836045 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 28 because replica has last-logged OpId of term: 27 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 142. I1017 23:33:30.836544 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 28 because replica has last-logged OpId of term: 27 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 142. Illegal state: not currently leaderI1017 23:33:30.837641 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 28. I1017 23:33:30.838028 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:30.838083 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. I1017 23:33:30.838273 21203 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.767s (election complete) I1017 23:33:30.838347 21203 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 FOLLOWER]: Leader election won for term 28 I1017 23:33:30.838740 21203 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:30.839215 21203 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 26.142, Last appended by leader: 142, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:30.841519 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 27 to 28, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 28 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:31.104300 21211 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:31.163203 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40127 I1017 23:33:31.163358 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 LEADER]: Received request to step down I1017 23:33:31.163439 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 LEADER]: Stepping down as leader of term 28 I1017 23:33:31.163481 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:31.163710 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 28.143, Last appended by leader: 143, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:31.163848 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:31.163887 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:31.163951 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 28 FOLLOWER]: Advancing to term 29 I1017 23:33:31.164007 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:32.441988 21232 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:32.496549 21247 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:32.496687 21247 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.082s (starting election) I1017 23:33:32.496785 21247 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:32.497326 21247 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:32.497550 21247 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:32.498616 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 29 candidate_status { last_received { term: 27 index: 143 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:32.498618 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 29 candidate_status { last_received { term: 27 index: 143 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:32.498800 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.460s (vote granted) I1017 23:33:32.498857 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 29 because replica has last-logged OpId of term: 28 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 143. I1017 23:33:32.498872 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 28. I1017 23:33:32.499642 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:32.499748 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. I1017 23:33:32.499876 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 29 because replica has last-logged OpId of term: 28 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 143. I1017 23:33:32.499997 21247 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.121s (election complete) I1017 23:33:32.500068 21247 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Leader pre-election won for term 29 I1017 23:33:32.500116 21247 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:32.500170 21247 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.612s (starting election) I1017 23:33:32.500224 21247 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 28 FOLLOWER]: Advancing to term 29 I1017 23:33:32.502230 21247 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:32.502709 21247 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:32.502925 21247 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:32.503484 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 29 candidate_status { last_received { term: 27 index: 143 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:32.503608 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 28 FOLLOWER]: Advancing to term 29 I1017 23:33:32.503690 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 4.500s (vote granted) I1017 23:33:32.503665 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 29 candidate_status { last_received { term: 27 index: 143 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:32.503830 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 29 because replica has last-logged OpId of term: 28 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 143. I1017 23:33:32.504398 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 29 because replica has last-logged OpId of term: 28 index: 143, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 143. I1017 23:33:32.505193 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 29. I1017 23:33:32.505638 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:32.505698 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. I1017 23:33:32.505751 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40130 I1017 23:33:32.505865 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Received request to transfer leadership I1017 23:33:32.505899 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:32.505901 21247 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.699s (election complete) I1017 23:33:32.505971 21247 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 FOLLOWER]: Leader election won for term 29 I1017 23:33:32.506281 21247 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:32.506566 21247 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 27.143, Last appended by leader: 143, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:32.508919 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 28 to 29, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 29 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } Illegal state: not currently leader W1017 23:33:32.781374 21257 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:32.840061 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35390 I1017 23:33:32.840229 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 LEADER]: Received request to step down I1017 23:33:32.840291 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 LEADER]: Stepping down as leader of term 29 I1017 23:33:32.840333 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:32.840610 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 29.144, Last appended by leader: 144, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:32.840756 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:32.840813 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:32.840890 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 29 FOLLOWER]: Advancing to term 30 I1017 23:33:32.840945 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:34.113898 21274 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:34.165696 21287 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:34.165858 21287 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.881s (starting election) I1017 23:33:34.165995 21287 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:34.166635 21287 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:34.166847 21287 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:34.167668 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 30 candidate_status { last_received { term: 28 index: 143 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:34.167945 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 30 because replica has last-logged OpId of term: 29 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 143. I1017 23:33:34.167949 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 30 candidate_status { last_received { term: 28 index: 143 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:34.168154 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.987s (vote granted) I1017 23:33:34.168231 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 29. I1017 23:33:34.168789 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 30 because replica has last-logged OpId of term: 29 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 143. I1017 23:33:34.168987 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:34.169052 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. I1017 23:33:34.169361 21287 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.195s (election complete) I1017 23:33:34.169481 21287 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Leader pre-election won for term 30 I1017 23:33:34.169564 21287 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:34.169656 21287 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.629s (starting election) I1017 23:33:34.169729 21287 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 29 FOLLOWER]: Advancing to term 30 I1017 23:33:34.171931 21287 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:34.172336 21287 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:34.173100 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 30 candidate_status { last_received { term: 28 index: 143 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:34.173357 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 30 because replica has last-logged OpId of term: 29 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 143. I1017 23:33:34.173936 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 30 because replica has last-logged OpId of term: 29 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 143. I1017 23:33:34.174075 21287 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:34.174790 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 30 candidate_status { last_received { term: 28 index: 143 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:34.174924 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 29 FOLLOWER]: Advancing to term 30 I1017 23:33:34.175004 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.563s (vote granted) I1017 23:33:34.176868 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 30. I1017 23:33:34.177310 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:34.177366 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. I1017 23:33:34.177608 21287 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.718s (election complete) I1017 23:33:34.177680 21287 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 FOLLOWER]: Leader election won for term 30 I1017 23:33:34.177989 21287 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:34.178246 21287 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 28.143, Last appended by leader: 143, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:34.179206 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35392 I1017 23:33:34.179334 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Received request to transfer leadership I1017 23:33:34.179384 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:34.180454 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 29 to 30, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 30 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } Illegal state: not currently leader W1017 23:33:34.451900 21299 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:34.512219 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40136 I1017 23:33:34.512380 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 LEADER]: Received request to step down I1017 23:33:34.512457 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 LEADER]: Stepping down as leader of term 30 I1017 23:33:34.512501 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:34.512749 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 30.144, Last appended by leader: 144, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:34.512887 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:34.512945 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:34.513031 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 30 FOLLOWER]: Advancing to term 31 I1017 23:33:34.513101 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:35.788406 21316 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:35.842378 21331 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:35.842463 21331 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.138s (starting election) I1017 23:33:35.842643 21331 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:35.843235 21331 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:35.843561 21331 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:35.844321 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 31 candidate_status { last_received { term: 29 index: 144 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:35.844549 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.560s (vote granted) I1017 23:33:35.844527 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 31 candidate_status { last_received { term: 29 index: 144 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:35.844621 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 30. I1017 23:33:35.844775 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 31 because replica has last-logged OpId of term: 30 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 144. I1017 23:33:35.845093 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:35.845186 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. I1017 23:33:35.845412 21331 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.841s (election complete) I1017 23:33:35.845527 21331 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Leader pre-election won for term 31 I1017 23:33:35.845614 21331 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:35.845614 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 31 because replica has last-logged OpId of term: 30 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 144. I1017 23:33:35.845695 21331 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.904s (starting election) I1017 23:33:35.845757 21331 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 30 FOLLOWER]: Advancing to term 31 I1017 23:33:35.847182 21331 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:35.847537 21331 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:35.847714 21331 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:35.848259 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 31 candidate_status { last_received { term: 29 index: 144 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:35.848372 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 30 FOLLOWER]: Advancing to term 31 I1017 23:33:35.848461 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 4.633s (vote granted) I1017 23:33:35.848522 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 31 candidate_status { last_received { term: 29 index: 144 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:35.848670 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 31 because replica has last-logged OpId of term: 30 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 144. I1017 23:33:35.849207 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 31 because replica has last-logged OpId of term: 30 index: 144, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 144. I1017 23:33:35.849908 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 31. I1017 23:33:35.849892 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40138 I1017 23:33:35.849994 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Received request to transfer leadership I1017 23:33:35.850044 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:35.850327 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:35.850394 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. I1017 23:33:35.850690 21331 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.717s (election complete) I1017 23:33:35.850770 21331 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 FOLLOWER]: Leader election won for term 31 I1017 23:33:35.851063 21331 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:35.851325 21331 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 29.144, Last appended by leader: 144, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:35.853826 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 30 to 31, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 31 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } Illegal state: not currently leader W1017 23:33:36.123507 21341 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:36.182006 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35398 I1017 23:33:36.182145 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 LEADER]: Received request to step down I1017 23:33:36.182196 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 LEADER]: Stepping down as leader of term 31 I1017 23:33:36.182251 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:36.182539 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 31.145, Last appended by leader: 145, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:36.182691 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:36.182739 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:36.182812 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 31 FOLLOWER]: Advancing to term 32 I1017 23:33:36.182875 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:37.455302 21362 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:37.514400 21379 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:37.514508 21379 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.689s (starting election) I1017 23:33:37.514606 21379 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:37.515008 21379 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:37.515209 21379 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:37.515954 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 32 candidate_status { last_received { term: 30 index: 144 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:37.516129 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 32 because replica has last-logged OpId of term: 31 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 144. I1017 23:33:37.516188 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 32 candidate_status { last_received { term: 30 index: 144 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:37.516346 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.785s (vote granted) I1017 23:33:37.516425 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 31. I1017 23:33:37.516841 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 32 because replica has last-logged OpId of term: 31 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 144. I1017 23:33:37.517055 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:37.517099 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. I1017 23:33:37.517185 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35401 I1017 23:33:37.517285 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Received request to transfer leadership I1017 23:33:37.517282 21379 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.014s (election complete) I1017 23:33:37.517320 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:37.517352 21379 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Leader pre-election won for term 32 I1017 23:33:37.517421 21379 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:37.517498 21379 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.538s (starting election) I1017 23:33:37.517581 21379 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 31 FOLLOWER]: Advancing to term 32 I1017 23:33:37.520000 21379 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:37.520612 21379 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:37.520823 21379 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) Illegal state: not currently leaderI1017 23:33:37.521412 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 32 candidate_status { last_received { term: 30 index: 144 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:37.521577 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 32 because replica has last-logged OpId of term: 31 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 144. I1017 23:33:37.521543 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 32 candidate_status { last_received { term: 30 index: 144 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:37.521672 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 31 FOLLOWER]: Advancing to term 32 I1017 23:33:37.521744 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.758s (vote granted) I1017 23:33:37.522090 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 32 because replica has last-logged OpId of term: 31 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 144. I1017 23:33:37.523242 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 32. I1017 23:33:37.523782 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:37.523839 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. I1017 23:33:37.524050 21379 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.008s (election complete) I1017 23:33:37.524129 21379 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 FOLLOWER]: Leader election won for term 32 I1017 23:33:37.524466 21379 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:37.524755 21379 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 30.144, Last appended by leader: 144, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:37.526932 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 31 to 32, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 32 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:37.790915 21387 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:37.853032 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40145 I1017 23:33:37.853199 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 LEADER]: Received request to step down I1017 23:33:37.853260 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 LEADER]: Stepping down as leader of term 32 I1017 23:33:37.853302 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:37.853533 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 32.145, Last appended by leader: 145, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:37.853682 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:37.853719 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:37.853785 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 32 FOLLOWER]: Advancing to term 33 I1017 23:33:37.853837 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:39.126417 21404 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:39.184252 21421 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:39.184331 21421 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.625s (starting election) I1017 23:33:39.184439 21421 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:39.184849 21421 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:39.185050 21421 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:39.185715 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 33 candidate_status { last_received { term: 31 index: 145 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:39.185894 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.613s (vote granted) I1017 23:33:39.185853 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 33 candidate_status { last_received { term: 31 index: 145 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:39.185958 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 32. I1017 23:33:39.186167 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 33 because replica has last-logged OpId of term: 32 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 145. I1017 23:33:39.186529 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:39.186556 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40147 I1017 23:33:39.186630 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. I1017 23:33:39.186661 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Received request to transfer leadership I1017 23:33:39.186697 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:39.186863 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 33 because replica has last-logged OpId of term: 32 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 145. I1017 23:33:39.186964 21421 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.041s (election complete) I1017 23:33:39.187065 21421 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Leader pre-election won for term 33 I1017 23:33:39.187119 21421 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:39.187173 21421 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.788s (starting election) I1017 23:33:39.187237 21421 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 32 FOLLOWER]: Advancing to term 33 I1017 23:33:39.188838 21421 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:39.189155 21421 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:39.189327 21421 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:39.189965 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 33 candidate_status { last_received { term: 31 index: 145 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:39.190076 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 32 FOLLOWER]: Advancing to term 33 I1017 23:33:39.190045 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 33 candidate_status { last_received { term: 31 index: 145 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:39.190158 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 1.887s (vote granted) I1017 23:33:39.190232 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 33 because replica has last-logged OpId of term: 32 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 145. I1017 23:33:39.190809 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 33 because replica has last-logged OpId of term: 32 index: 145, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 145. I1017 23:33:39.192044 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 33. I1017 23:33:39.192551 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:39.192613 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Illegal state: not currently leaderI1017 23:33:39.192891 21421 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.943s (election complete) I1017 23:33:39.192966 21421 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 FOLLOWER]: Leader election won for term 33 I1017 23:33:39.193308 21421 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:39.193603 21421 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 31.145, Last appended by leader: 145, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:39.195906 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 32 to 33, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 33 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:39.461247 21429 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:39.520928 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35407 I1017 23:33:39.521073 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 LEADER]: Received request to step down I1017 23:33:39.521126 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 LEADER]: Stepping down as leader of term 33 I1017 23:33:39.521158 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:39.521363 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 33.146, Last appended by leader: 146, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:39.521533 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:39.521586 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:39.521659 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 33 FOLLOWER]: Advancing to term 34 I1017 23:33:39.521720 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:40.797794 21446 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:40.854341 21463 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:40.854423 21463 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.789s (starting election) I1017 23:33:40.854549 21463 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:40.854935 21463 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:40.855093 21463 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:40.855883 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 34 candidate_status { last_received { term: 32 index: 145 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:40.856088 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 34 because replica has last-logged OpId of term: 33 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 145. I1017 23:33:40.856106 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 34 candidate_status { last_received { term: 32 index: 145 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:40.856266 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.999s (vote granted) I1017 23:33:40.856338 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 33. I1017 23:33:40.856894 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:40.856953 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. I1017 23:33:40.857162 21463 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.693s (election complete) I1017 23:33:40.857321 21463 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Leader pre-election won for term 34 I1017 23:33:40.857306 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 34 because replica has last-logged OpId of term: 33 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 145. I1017 23:33:40.857448 21463 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:40.857537 21463 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.515s (starting election) I1017 23:33:40.857590 21463 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 33 FOLLOWER]: Advancing to term 34 I1017 23:33:40.858186 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35409 I1017 23:33:40.858294 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Received request to transfer leadership I1017 23:33:40.858341 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:40.859529 21463 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:40.859923 21463 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:40.860286 21463 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:40.860564 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 34 candidate_status { last_received { term: 32 index: 145 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:40.860723 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 34 because replica has last-logged OpId of term: 33 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 145. I1017 23:33:40.860990 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 34 candidate_status { last_received { term: 32 index: 145 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:40.861116 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 33 FOLLOWER]: Advancing to term 34 I1017 23:33:40.861192 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.413s (vote granted) I1017 23:33:40.861881 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 34 because replica has last-logged OpId of term: 33 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 145. I1017 23:33:40.862905 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 34. Illegal state: not currently leader I1017 23:33:40.863324 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:40.863371 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. I1017 23:33:40.863541 21463 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.042s (election complete) I1017 23:33:40.863636 21463 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 FOLLOWER]: Leader election won for term 34 I1017 23:33:40.864046 21463 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:40.864321 21463 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 32.145, Last appended by leader: 145, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:40.866686 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 33 to 34, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 34 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:41.131738 21471 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:41.193035 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40153 I1017 23:33:41.193178 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 LEADER]: Received request to step down I1017 23:33:41.193241 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 LEADER]: Stepping down as leader of term 34 I1017 23:33:41.193289 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:41.193539 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 34.146, Last appended by leader: 146, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:41.193697 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:41.193759 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:41.193859 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 34 FOLLOWER]: Advancing to term 35 I1017 23:33:41.193933 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:42.469012 21492 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:42.523339 21507 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:42.523423 21507 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.099s (starting election) I1017 23:33:42.523547 21507 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:42.523972 21507 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:42.524164 21507 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:42.524960 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 35 candidate_status { last_received { term: 33 index: 146 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:42.525122 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 4.104s (vote granted) I1017 23:33:42.525106 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 35 candidate_status { last_received { term: 33 index: 146 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:42.525190 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 34. I1017 23:33:42.525351 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 35 because replica has last-logged OpId of term: 34 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 146. I1017 23:33:42.525835 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:42.525935 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. I1017 23:33:42.526079 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 35 because replica has last-logged OpId of term: 34 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 146. I1017 23:33:42.526245 21507 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.933s (election complete) I1017 23:33:42.526343 21507 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Leader pre-election won for term 35 I1017 23:33:42.526392 21507 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:42.526469 21507 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.642s (starting election) I1017 23:33:42.526525 21507 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 34 FOLLOWER]: Advancing to term 35 I1017 23:33:42.527962 21507 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:42.528275 21507 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:42.528419 21507 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:42.529155 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 35 candidate_status { last_received { term: 33 index: 146 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:42.529163 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 35 candidate_status { last_received { term: 33 index: 146 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:42.529263 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 34 FOLLOWER]: Advancing to term 35 I1017 23:33:42.529317 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 35 because replica has last-logged OpId of term: 34 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 146. I1017 23:33:42.529325 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.257s (vote granted) I1017 23:33:42.529886 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 35 because replica has last-logged OpId of term: 34 index: 146, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 146. I1017 23:33:42.530786 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 35. I1017 23:33:42.531302 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:42.531360 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. I1017 23:33:42.531558 21507 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.629s (election complete) I1017 23:33:42.531536 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40156 I1017 23:33:42.531638 21507 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 FOLLOWER]: Leader election won for term 35 I1017 23:33:42.531667 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Received request to transfer leadership I1017 23:33:42.531724 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:42.532057 21507 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:42.532297 21507 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 33.146, Last appended by leader: 146, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:42.534585 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 34 to 35, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 35 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } Illegal state: not currently leader W1017 23:33:42.806967 21517 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:42.866534 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35416 I1017 23:33:42.866683 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 LEADER]: Received request to step down I1017 23:33:42.866734 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 LEADER]: Stepping down as leader of term 35 I1017 23:33:42.866766 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:42.866984 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 35.147, Last appended by leader: 147, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:42.867141 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:42.867189 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:42.867262 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 35 FOLLOWER]: Advancing to term 36 I1017 23:33:42.867323 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:44.138643 21534 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:44.194936 21551 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:44.195020 21551 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.093s (starting election) I1017 23:33:44.195168 21551 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:44.195585 21551 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:44.195791 21551 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:44.196522 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 36 candidate_status { last_received { term: 34 index: 146 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:44.196702 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 36 because replica has last-logged OpId of term: 35 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 146. I1017 23:33:44.196646 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 36 candidate_status { last_received { term: 34 index: 146 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:44.196791 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 4.907s (vote granted) I1017 23:33:44.196866 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 35. I1017 23:33:44.197279 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 36 because replica has last-logged OpId of term: 35 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 146. I1017 23:33:44.197520 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:44.197582 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. I1017 23:33:44.197782 21551 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.248s (election complete) I1017 23:33:44.197876 21551 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Leader pre-election won for term 36 I1017 23:33:44.197923 21551 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:44.197983 21551 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.875s (starting election) I1017 23:33:44.198041 21551 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 35 FOLLOWER]: Advancing to term 36 I1017 23:33:44.199044 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35418 I1017 23:33:44.199162 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Received request to transfer leadership I1017 23:33:44.199204 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:44.200314 21551 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:44.200744 21551 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:44.200932 21551 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:44.201557 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 36 candidate_status { last_received { term: 34 index: 146 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:44.201795 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 36 because replica has last-logged OpId of term: 35 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 146. I1017 23:33:44.201802 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 36 candidate_status { last_received { term: 34 index: 146 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:44.201922 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 35 FOLLOWER]: Advancing to term 36 I1017 23:33:44.201992 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.938s (vote granted) I1017 23:33:44.202437 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 36 because replica has last-logged OpId of term: 35 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 146. Illegal state: not currently leader I1017 23:33:44.203924 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 36. I1017 23:33:44.204342 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:44.204393 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. I1017 23:33:44.204617 21551 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.961s (election complete) I1017 23:33:44.204691 21551 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 FOLLOWER]: Leader election won for term 36 I1017 23:33:44.205056 21551 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:44.205348 21551 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 34.146, Last appended by leader: 146, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:44.207639 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 35 to 36, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 36 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:44.470149 21559 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:44.530459 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: ABRUPT from {username='jenkins'} at 127.18.62.195:40162 I1017 23:33:44.530623 19960 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 LEADER]: Received request to step down I1017 23:33:44.530676 19960 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 LEADER]: Stepping down as leader of term 36 I1017 23:33:44.530709 19960 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:44.530906 19960 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 36.147, Last appended by leader: 147, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:44.531054 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:44.531105 19960 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 -> Peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726): Closing peer: 0ecfe08279804cb0888b118fc4f87243 I1017 23:33:44.531179 19960 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 36 FOLLOWER]: Advancing to term 37 I1017 23:33:44.531241 19960 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:45.803367 21576 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:45.863291 19960 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" mode: GRACEFUL from {username='jenkins'} at 127.18.62.195:40164 I1017 23:33:45.863432 19960 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Received request to transfer leadership I1017 23:33:45.863503 19960 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Rejecting request to transer leadership while not leader Illegal state: not currently leader I1017 23:33:45.868839 21593 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:45.868914 21593 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.991s (starting election) I1017 23:33:45.869015 21593 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:45.869415 21593 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:45.869637 21593 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 pre-election: Requesting pre-vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:45.870331 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 37 candidate_status { last_received { term: 35 index: 147 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:45.870337 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 37 candidate_status { last_received { term: 35 index: 147 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" is_pre_election: true I1017 23:33:45.870504 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.979s (vote granted) I1017 23:33:45.870577 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 37 because replica has last-logged OpId of term: 36 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 147. I1017 23:33:45.870573 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 36. I1017 23:33:45.871088 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:45.871173 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. I1017 23:33:45.871286 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 pre-election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 37 because replica has last-logged OpId of term: 36 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 147. I1017 23:33:45.871400 21593 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.069s (election complete) I1017 23:33:45.871505 21593 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Leader pre-election won for term 37 I1017 23:33:45.871588 21593 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:45.871647 21593 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 1.596s (starting election) I1017 23:33:45.871708 21593 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 36 FOLLOWER]: Advancing to term 37 I1017 23:33:45.873194 21593 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:45.873528 21593 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:45.873703 21593 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 election: Requesting vote from peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672) I1017 23:33:45.874251 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 37 candidate_status { last_received { term: 35 index: 147 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:45.874364 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 36 FOLLOWER]: Advancing to term 37 I1017 23:33:45.874451 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 3.214s (vote granted) I1017 23:33:45.874425 19960 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "0ecfe08279804cb0888b118fc4f87243" candidate_term: 37 candidate_status { last_received { term: 35 index: 147 } } ignore_live_leader: false dest_uuid: "1ad000b8f6704a80bc4330f4e0c41996" I1017 23:33:45.874598 19960 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 37 because replica has last-logged OpId of term: 36 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 147. I1017 23:33:45.875095 19646 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 election: Vote denied by peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ecfe08279804cb0888b118fc4f87243 for term 37 because replica has last-logged OpId of term: 36 index: 147, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 147. I1017 23:33:45.875852 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ecfe08279804cb0888b118fc4f87243 in term 37. I1017 23:33:45.876332 19644 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:45.876391 19644 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. I1017 23:33:45.876600 21593 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 2.199s (election complete) I1017 23:33:45.876695 21593 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 FOLLOWER]: Leader election won for term 37 I1017 23:33:45.877010 21593 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 LEADER]: Becoming Leader. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:45.877475 21593 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 35.147, Last appended by leader: 147, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:45.879971 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 reported cstate change: term changed from 36 to 37, leader changed from 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195) to 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193). New cstate: current_term: 37 leader_uuid: "0ecfe08279804cb0888b118fc4f87243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: UNKNOWN } } } W1017 23:33:46.138046 21601 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:46.198333 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: ABRUPT from {username='jenkins'} at 127.18.62.193:35424 I1017 23:33:46.198510 19711 raft_consensus.cc:568] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 LEADER]: Received request to step down I1017 23:33:46.198572 19711 raft_consensus.cc:2887] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 LEADER]: Stepping down as leader of term 37 I1017 23:33:46.198621 19711 raft_consensus.cc:704] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 0ecfe08279804cb0888b118fc4f87243, State: Running, Role: LEADER I1017 23:33:46.198849 19711 consensus_queue.cc:229] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 37.148, Last appended by leader: 148, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:46.199002 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195:55672): Closing peer: 1ad000b8f6704a80bc4330f4e0c41996 I1017 23:33:46.199062 19711 consensus_peers.cc:469] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 -> Peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Closing peer: b01d528fd3c74eb5b42b8d4888591ed2 I1017 23:33:46.199173 19711 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 37 FOLLOWER]: Advancing to term 38 I1017 23:33:46.199251 19711 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243: Snoozing failure detection for 3.000s (explicit stepdown request) W1017 23:33:47.473224 21622 flags.cc:406] Enabled unsafe flag: --never_fsync=true I1017 23:33:47.532428 21639 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1017 23:33:47.532529 21639 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.547s (starting election) I1017 23:33:47.532631 21639 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:47.533021 21639 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 pre-election: Requesting pre-vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) I1017 23:33:47.533223 21639 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 pre-election: Requesting pre-vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:47.533895 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 38 candidate_status { last_received { term: 36 index: 147 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" is_pre_election: true I1017 23:33:47.534049 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 38 candidate_status { last_received { term: 36 index: 147 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" is_pre_election: true I1017 23:33:47.534183 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 38 because replica has last-logged OpId of term: 37 index: 148, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 147. I1017 23:33:47.534193 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.125s (vote granted) I1017 23:33:47.534270 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 37. I1017 23:33:47.534904 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 pre-election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:47.534914 19711 tablet_service.cc:1146] Received LeaderStepDown RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" dest_uuid: "0ecfe08279804cb0888b118fc4f87243" mode: GRACEFUL from {username='jenkins'} at 127.18.62.193:35427 I1017 23:33:47.534965 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. I1017 23:33:47.535045 19711 raft_consensus.cc:583] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 38 FOLLOWER]: Received request to transfer leadership I1017 23:33:47.535096 19711 raft_consensus.cc:591] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader I1017 23:33:47.535162 21639 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.805s (election complete) I1017 23:33:47.535254 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 pre-election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 38 because replica has last-logged OpId of term: 37 index: 148, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 147. I1017 23:33:47.535388 21639 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Leader pre-election won for term 38 I1017 23:33:47.535449 21639 raft_consensus.cc:471] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1017 23:33:47.535537 21639 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 1.916s (starting election) I1017 23:33:47.535604 21639 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 37 FOLLOWER]: Advancing to term 38 I1017 23:33:47.538426 21639 raft_consensus.cc:493] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:47.539062 21639 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 election: Requesting vote from peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726) Illegal state: not currently leader I1017 23:33:47.539520 21639 leader_election.cc:251] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 election: Requesting vote from peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:47.539784 19711 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 38 candidate_status { last_received { term: 36 index: 147 } } ignore_live_leader: false dest_uuid: "0ecfe08279804cb0888b118fc4f87243" I1017 23:33:47.539932 19711 raft_consensus.cc:2299] T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 38 because replica has last-logged OpId of term: 37 index: 148, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 147. I1017 23:33:47.540215 19839 tablet_service.cc:1001] Received RequestConsensusVote() RPC: tablet_id: "9b4b2dea960941bcb38197b51c55baf4" candidate_uuid: "1ad000b8f6704a80bc4330f4e0c41996" candidate_term: 38 candidate_status { last_received { term: 36 index: 147 } } ignore_live_leader: false dest_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" I1017 23:33:47.540356 19839 raft_consensus.cc:2892] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 37 FOLLOWER]: Advancing to term 38 I1017 23:33:47.540427 19839 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2: Snoozing failure detection for 2.258s (vote granted) I1017 23:33:47.540464 19893 leader_election.cc:401] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 election: Vote denied by peer 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193:39726). Message: Invalid argument: T 9b4b2dea960941bcb38197b51c55baf4 P 0ecfe08279804cb0888b118fc4f87243 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 1ad000b8f6704a80bc4330f4e0c41996 for term 38 because replica has last-logged OpId of term: 37 index: 148, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 147. I1017 23:33:47.541903 19839 raft_consensus.cc:2352] T 9b4b2dea960941bcb38197b51c55baf4 P b01d528fd3c74eb5b42b8d4888591ed2 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ad000b8f6704a80bc4330f4e0c41996 in term 38. I1017 23:33:47.542423 19893 leader_election.cc:387] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 election: Vote granted by peer b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) I1017 23:33:47.542507 19893 leader_election.cc:278] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. I1017 23:33:47.542702 21639 raft_consensus.cc:2840] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996: Snoozing failure detection for 2.105s (election complete) I1017 23:33:47.542760 21639 raft_consensus.cc:2646] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 38 FOLLOWER]: Leader election won for term 38 I1017 23:33:47.543037 21639 raft_consensus.cc:666] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [term 38 LEADER]: Becoming Leader. State: Replica: 1ad000b8f6704a80bc4330f4e0c41996, State: Running, Role: LEADER I1017 23:33:47.543335 21639 consensus_queue.cc:206] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 36.147, Last appended by leader: 147, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } } I1017 23:33:47.545567 19590 catalog_manager.cc:3758] T 9b4b2dea960941bcb38197b51c55baf4 P 1ad000b8f6704a80bc4330f4e0c41996 reported cstate change: term changed from 37 to 38, leader changed from 0ecfe08279804cb0888b118fc4f87243 (127.18.62.193) to 1ad000b8f6704a80bc4330f4e0c41996 (127.18.62.195). New cstate: current_term: 38 leader_uuid: "1ad000b8f6704a80bc4330f4e0c41996" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ecfe08279804cb0888b118fc4f87243" member_type: VOTER last_known_addr { host: "127.18.62.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b01d528fd3c74eb5b42b8d4888591ed2" member_type: VOTER last_known_addr { host: "127.18.62.194" port: 38185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ad000b8f6704a80bc4330f4e0c41996" member_type: VOTER last_known_addr { host: "127.18.62.195" port: 55672 } health_report { overall_health: HEALTHY } } } W1017 23:33:47.771695 20038 meta_cache.cc:216] Tablet 9b4b2dea960941bcb38197b51c55baf4: Replica b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185) has failed: Timed out: Write RPC to 127.18.62.194:38185 timed out after 60.000s (SENT) W1017 23:33:47.772014 20038 batcher.cc:348] Timed out: Failed to write batch of 1 ops to tablet 9b4b2dea960941bcb38197b51c55baf4 after 1 attempt(s): Failed to write to server: b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Write RPC to 127.18.62.194:38185 timed out after 60.000s (SENT) F1017 23:33:47.772820 20042 test_workload.cc:202] Timed out: Failed to write batch of 1 ops to tablet 9b4b2dea960941bcb38197b51c55baf4 after 1 attempt(s): Failed to write to server: b01d528fd3c74eb5b42b8d4888591ed2 (127.18.62.194:38185): Write RPC to 127.18.62.194:38185 timed out after 60.000s (SENT) *** Check failure stack trace: *** *** Aborted at 1539844427 (unix time) try "date -d @1539844427" if you are using GNU date *** PC: @ 0x3c74632625 __GI_raise *** SIGABRT (@0x452000048fb) received by PID 18683 (TID 0x7f13ebe5b700) from PID 18683; stack trace: *** @ 0x3c74a0f710 (unknown) at ??:0 @ 0x3c74632625 __GI_raise at ??:0 @ 0x3c74633e05 __GI_abort at ??:0 @ 0x7f13fd43da29 (unknown) at ??:0 @ 0x7f13fd43f31d (unknown) at ??:0 @ 0x7f13fd4411dd (unknown) at ??:0 @ 0x7f13fd43ee59 (unknown) at ??:0 @ 0x7f13fd441c7f (unknown) at ??:0 @ 0x7f1412f7ba6e (unknown) at ??:0 @ 0x3c796b6470 (unknown) at ??:0 @ 0x3c74a079d1 start_thread at ??:0 @ 0x3c746e88fd clone at ??:0