[==========] Running 8 tests from 2 test cases. [----------] Global test environment set-up. [----------] 4 tests from DiskFailure/DiskFailureITest [ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/0 WARNING: Logging before InitGoogleLogging() is written to STDERR I0720 00:42:27.523995 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-4 --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:37585 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:27.716732 240 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:27.716934 240 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:27.719404 240 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:27.719452 240 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:42:27.719488 240 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:27.719521 240 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:42:27.719569 240 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:42:27.724750 240 master_main.cc:98] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:37585 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:27.731811 240 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 35016us I0720 00:42:27.733191 240 fs_manager.cc:263] Metadata directory not provided I0720 00:42:27.733253 240 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:27.733604 240 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:27.733669 240 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:27.736135 240 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.736657 240 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.737007 240 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.737310 240 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-3/instance: uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.737629 240 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-4/instance: uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.737946 240 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.748802 240 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.008s sys 0.001s I0720 00:42:27.750160 240 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:27.750938 240 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:27.751505 240 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0720 00:42:27.751569 240 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/wal uuid: "d589d9ed25814d82889b1dbe3cdb2b70" format_stamp: "Formatted at 2019-07-20 00:42:27 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:27.751700 240 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:27.781916 240 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:27.797724 240 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:37585 I0720 00:42:27.798810 240 webserver.cc:374] Webserver started at http://127.0.58.190:40825/ using document root and password file I0720 00:42:27.799484 240 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:42:27.805037 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 240 I0720 00:42:27.806195 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37585 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:27.816464 303 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70: Bootstrap starting. I0720 00:42:27.819115 303 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70: No blocks or log segments found. Creating new log. I0720 00:42:27.820008 303 log.cc:525] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70: Log is configured to *not* fsync() on all Append() calls I0720 00:42:27.821822 303 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70: No bootstrap required, opened a new log I0720 00:42:27.831640 303 raft_consensus.cc:340] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } I0720 00:42:27.831940 303 raft_consensus.cc:366] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:27.832074 303 raft_consensus.cc:705] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d589d9ed25814d82889b1dbe3cdb2b70, State: Initialized, Role: FOLLOWER I0720 00:42:27.832640 303 consensus_queue.cc:229] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } I0720 00:42:27.832934 303 raft_consensus.cc:380] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:42:27.833353 303 raft_consensus.cc:472] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:42:27.833493 303 raft_consensus.cc:2890] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:27.835820 303 raft_consensus.cc:494] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } I0720 00:42:27.836416 303 leader_election.cc:310] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d589d9ed25814d82889b1dbe3cdb2b70; no voters: I0720 00:42:27.837196 303 leader_election.cc:296] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:42:27.837400 306 raft_consensus.cc:2644] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:27.838255 306 raft_consensus.cc:667] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [term 1 LEADER]: Becoming Leader. State: Replica: d589d9ed25814d82889b1dbe3cdb2b70, State: Running, Role: LEADER I0720 00:42:27.838644 306 consensus_queue.cc:206] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } I0720 00:42:27.840660 308 sys_catalog.cc:340] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d589d9ed25814d82889b1dbe3cdb2b70. Latest consensus state: current_term: 1 leader_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } } I0720 00:42:27.841789 308 sys_catalog.cc:343] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: This master's current role is: LEADER I0720 00:42:27.840690 307 sys_catalog.cc:340] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } } I0720 00:42:27.842118 307 sys_catalog.cc:343] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: This master's current role is: LEADER I0720 00:42:27.842540 303 sys_catalog.cc:340] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d589d9ed25814d82889b1dbe3cdb2b70" member_type: VOTER } } I0720 00:42:27.842725 303 sys_catalog.cc:343] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: This master's current role is: LEADER I0720 00:42:27.844715 303 sys_catalog.cc:428] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70 [sys.catalog]: configured and running, proceeding with master startup. I0720 00:42:27.844923 310 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:42:27.856562 310 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:42:27.972343 310 catalog_manager.cc:959] Generated new certificate authority record I0720 00:42:27.973628 310 catalog_manager.cc:1084] Loading token signing keys... I0720 00:42:27.987416 310 catalog_manager.cc:4371] T 00000000000000000000000000000000 P d589d9ed25814d82889b1dbe3cdb2b70: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:28.019904 304 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:28.020187 304 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:28.023339 304 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:28.023413 304 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:28.023569 304 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:42:28.034287 304 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37585 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:28.045603 304 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 35516us I0720 00:42:28.046612 304 fs_manager.cc:263] Metadata directory not provided I0720 00:42:28.046689 304 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:28.047102 304 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:28.047183 304 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:28.050200 304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.050875 304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.051316 304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.051739 304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3/instance: uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.052165 304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4/instance: uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.052599 304 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.071383 304 fs_manager.cc:503] Time spent creating directory manager: real 0.018s user 0.019s sys 0.001s I0720 00:42:28.072856 304 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:28.073428 304 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:28.074342 304 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.002s I0720 00:42:28.074424 304 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.074573 304 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:28.124408 304 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:28.126555 304 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:28.126632 304 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:28.162719 304 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:34255 I0720 00:42:28.163852 304 webserver.cc:374] Webserver started at http://127.0.58.129:45131/ using document root and password file I0720 00:42:28.164510 304 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:42:28.169827 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 304 I0720 00:42:28.170892 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37585 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:28.189608 437 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37585 I0720 00:42:28.189849 437 heartbeater.cc:405] Registering TS with master... I0720 00:42:28.190415 437 heartbeater.cc:454] Master 127.0.58.190:37585 requested a full tablet report, sending... I0720 00:42:28.192523 270 ts_manager.cc:161] Registered new tserver with Master: 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) I0720 00:42:28.195080 270 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:57305 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:28.341328 441 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:28.341531 441 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:28.343665 441 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:28.343729 441 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:28.343840 441 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:42:28.347577 441 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37585 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:28.356029 441 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 35516us I0720 00:42:28.356770 441 fs_manager.cc:263] Metadata directory not provided I0720 00:42:28.356829 441 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:28.357136 441 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:28.357224 441 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:28.359530 441 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.360064 441 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.360417 441 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.360728 441 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-3/instance: uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.361047 441 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-4/instance: uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.361364 441 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.371486 441 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s I0720 00:42:28.372843 441 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:28.373332 441 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:28.373806 441 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.002s sys 0.001s I0720 00:42:28.373876 441 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/wal uuid: "4deaf10768b24896bcd716f79c982b0c" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.374004 441 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:28.395809 441 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:28.397343 441 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:28.397420 441 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:28.424296 441 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:38913 I0720 00:42:28.425403 441 webserver.cc:374] Webserver started at http://127.0.58.130:35607/ using document root and password file I0720 00:42:28.426012 441 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:42:28.433329 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 441 I0720 00:42:28.434180 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37585 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:28.449538 564 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37585 I0720 00:42:28.449738 564 heartbeater.cc:405] Registering TS with master... I0720 00:42:28.450193 564 heartbeater.cc:454] Master 127.0.58.190:37585 requested a full tablet report, sending... I0720 00:42:28.451231 270 ts_manager.cc:161] Registered new tserver with Master: 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913) I0720 00:42:28.452033 270 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:48641 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:28.594501 567 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:28.594709 567 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:28.596686 567 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:28.596735 567 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:28.596864 567 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:42:28.601166 567 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37585 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:28.609218 567 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 35516us I0720 00:42:28.609925 567 fs_manager.cc:263] Metadata directory not provided I0720 00:42:28.609977 567 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:28.610321 567 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:28.610379 567 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:28.612512 567 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.613061 567 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.613400 567 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.613711 567 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-3/instance: uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.614038 567 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-4/instance: uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.614393 567 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.624811 567 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.006s sys 0.002s I0720 00:42:28.626133 567 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:28.626610 567 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:28.627064 567 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0720 00:42:28.627125 567 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/wal uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:28.627239 567 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:28.663393 567 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:28.664908 567 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:28.664973 567 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:28.691555 567 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:34777 I0720 00:42:28.692550 567 webserver.cc:374] Webserver started at http://127.0.58.131:42101/ using document root and password file I0720 00:42:28.693166 567 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:42:28.697638 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 567 I0720 00:42:28.708068 690 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37585 I0720 00:42:28.708253 690 heartbeater.cc:405] Registering TS with master... I0720 00:42:28.708662 690 heartbeater.cc:454] Master 127.0.58.190:37585 requested a full tablet report, sending... I0720 00:42:28.709669 270 ts_manager.cc:161] Registered new tserver with Master: 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:28.711449 270 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:49125 I0720 00:42:28.716976 234 external_mini_cluster.cc:628] 3 TS(s) registered with all masters I0720 00:42:28.731972 234 test_util.cc:212] Using random seed: 504623145 I0720 00:42:28.750273 270 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41458: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff" } partition_schema { range_schema { columns { name: "key" } } } I0720 00:42:28.785899 372 tablet_service.cc:1022] Processing CreateTablet for tablet 34dd0711864545e1beb62517f20f557e (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:28.785581 370 tablet_service.cc:1022] Processing CreateTablet for tablet 6da351b3ad134fc78c052d34aafdcbc0 (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:28.785295 373 tablet_service.cc:1022] Processing CreateTablet for tablet ea63d074a81e47c7910a7097efab544e (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:28.785755 371 tablet_service.cc:1022] Processing CreateTablet for tablet 8668997e728c430da205e7d36533f74a (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:28.788430 627 tablet_service.cc:1022] Processing CreateTablet for tablet 7cab46396b7e471f89b67a9bee891c6a (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:28.789369 374 tablet_service.cc:1022] Processing CreateTablet for tablet 7cab46396b7e471f89b67a9bee891c6a (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:28.790501 626 tablet_service.cc:1022] Processing CreateTablet for tablet ea63d074a81e47c7910a7097efab544e (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:28.794448 625 tablet_service.cc:1022] Processing CreateTablet for tablet 34dd0711864545e1beb62517f20f557e (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:28.795454 624 tablet_service.cc:1022] Processing CreateTablet for tablet 8668997e728c430da205e7d36533f74a (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:28.796439 623 tablet_service.cc:1022] Processing CreateTablet for tablet 6da351b3ad134fc78c052d34aafdcbc0 (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:28.796945 501 tablet_service.cc:1022] Processing CreateTablet for tablet 7cab46396b7e471f89b67a9bee891c6a (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:28.802675 500 tablet_service.cc:1022] Processing CreateTablet for tablet ea63d074a81e47c7910a7097efab544e (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:28.805269 499 tablet_service.cc:1022] Processing CreateTablet for tablet 34dd0711864545e1beb62517f20f557e (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:28.806308 497 tablet_service.cc:1022] Processing CreateTablet for tablet 6da351b3ad134fc78c052d34aafdcbc0 (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:28.807379 498 tablet_service.cc:1022] Processing CreateTablet for tablet 8668997e728c430da205e7d36533f74a (table=test-workload [id=e0395db4648044fc9c454c18dec46a8e]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:28.827860 716 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:28.828202 717 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:28.828675 714 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:28.829684 718 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: Bootstrap starting. I0720 00:42:28.830044 715 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:28.835461 714 tablet_bootstrap.cc:590] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: No blocks or log segments found. Creating new log. I0720 00:42:28.836927 716 tablet_bootstrap.cc:590] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: No blocks or log segments found. Creating new log. I0720 00:42:28.839531 715 tablet_bootstrap.cc:590] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: No blocks or log segments found. Creating new log. I0720 00:42:28.840284 722 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c: Bootstrap starting. I0720 00:42:28.840749 721 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:28.841804 723 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c: Bootstrap starting. I0720 00:42:28.842720 715 log.cc:525] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Log is configured to *not* fsync() on all Append() calls I0720 00:42:28.843056 719 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c: Bootstrap starting. I0720 00:42:28.845160 720 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: Bootstrap starting. I0720 00:42:28.845165 717 tablet_bootstrap.cc:590] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: No blocks or log segments found. Creating new log. I0720 00:42:28.847393 718 tablet_bootstrap.cc:590] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: No blocks or log segments found. Creating new log. I0720 00:42:28.847484 721 tablet_bootstrap.cc:590] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: No blocks or log segments found. Creating new log. I0720 00:42:28.850509 723 tablet_bootstrap.cc:590] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c: No blocks or log segments found. Creating new log. I0720 00:42:28.851302 715 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: No bootstrap required, opened a new log I0720 00:42:28.851524 722 tablet_bootstrap.cc:590] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c: No blocks or log segments found. Creating new log. I0720 00:42:28.851616 718 log.cc:525] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: Log is configured to *not* fsync() on all Append() calls I0720 00:42:28.852149 715 ts_tablet_manager.cc:1059] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s I0720 00:42:28.853610 719 tablet_bootstrap.cc:590] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c: No blocks or log segments found. Creating new log. I0720 00:42:28.854354 720 tablet_bootstrap.cc:590] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: No blocks or log segments found. Creating new log. I0720 00:42:28.854467 716 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: No bootstrap required, opened a new log I0720 00:42:28.855888 716 ts_tablet_manager.cc:1059] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s I0720 00:42:28.858673 714 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: No bootstrap required, opened a new log I0720 00:42:28.858806 714 ts_tablet_manager.cc:1059] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s I0720 00:42:28.859329 727 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc: Bootstrap starting. I0720 00:42:28.861402 730 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Bootstrap starting. I0720 00:42:28.863165 717 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: No bootstrap required, opened a new log I0720 00:42:28.863282 717 ts_tablet_manager.cc:1059] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s I0720 00:42:28.869271 718 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: No bootstrap required, opened a new log I0720 00:42:28.869659 718 ts_tablet_manager.cc:1059] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.002s I0720 00:42:28.871022 729 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc: Bootstrap starting. I0720 00:42:28.873028 721 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: No bootstrap required, opened a new log I0720 00:42:28.873149 721 ts_tablet_manager.cc:1059] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s I0720 00:42:28.874230 737 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Bootstrap starting. I0720 00:42:28.874429 734 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc: Bootstrap starting. I0720 00:42:28.874554 722 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c: No bootstrap required, opened a new log I0720 00:42:28.875059 719 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c: No bootstrap required, opened a new log I0720 00:42:28.875907 719 ts_tablet_manager.cc:1059] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s I0720 00:42:28.875905 723 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c: No bootstrap required, opened a new log I0720 00:42:28.876097 722 ts_tablet_manager.cc:1059] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.001s I0720 00:42:28.876960 723 ts_tablet_manager.cc:1059] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s I0720 00:42:28.877075 720 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: No bootstrap required, opened a new log I0720 00:42:28.877295 720 ts_tablet_manager.cc:1059] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.004s I0720 00:42:28.879299 730 tablet_bootstrap.cc:590] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc: No blocks or log segments found. Creating new log. I0720 00:42:28.879781 729 tablet_bootstrap.cc:590] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc: No blocks or log segments found. Creating new log. I0720 00:42:28.880281 727 tablet_bootstrap.cc:590] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc: No blocks or log segments found. Creating new log. I0720 00:42:28.881559 730 log.cc:525] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Log is configured to *not* fsync() on all Append() calls I0720 00:42:28.882200 737 tablet_bootstrap.cc:590] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: No blocks or log segments found. Creating new log. I0720 00:42:28.883280 734 tablet_bootstrap.cc:590] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc: No blocks or log segments found. Creating new log. I0720 00:42:28.891350 729 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc: No bootstrap required, opened a new log I0720 00:42:28.891655 729 ts_tablet_manager.cc:1059] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s I0720 00:42:28.893626 730 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc: No bootstrap required, opened a new log I0720 00:42:28.896806 730 ts_tablet_manager.cc:1059] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.010s I0720 00:42:28.897742 737 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: No bootstrap required, opened a new log I0720 00:42:28.897902 737 ts_tablet_manager.cc:1059] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s I0720 00:42:28.900490 727 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc: No bootstrap required, opened a new log I0720 00:42:28.900614 727 ts_tablet_manager.cc:1059] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.008s I0720 00:42:28.904146 734 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc: No bootstrap required, opened a new log I0720 00:42:28.906471 734 ts_tablet_manager.cc:1059] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.005s I0720 00:42:28.918416 717 raft_consensus.cc:340] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.918999 717 raft_consensus.cc:366] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.919081 717 raft_consensus.cc:705] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:28.919850 717 consensus_queue.cc:229] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.920538 716 raft_consensus.cc:340] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.920815 716 raft_consensus.cc:366] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.920876 716 raft_consensus.cc:705] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:28.921104 716 consensus_queue.cc:229] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.921562 714 raft_consensus.cc:340] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:28.921775 714 raft_consensus.cc:366] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.921830 714 raft_consensus.cc:705] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:28.922042 714 consensus_queue.cc:229] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:28.923880 721 raft_consensus.cc:340] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.924155 721 raft_consensus.cc:366] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.924240 721 raft_consensus.cc:705] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:28.924477 721 consensus_queue.cc:229] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.928076 715 raft_consensus.cc:340] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.928285 715 raft_consensus.cc:366] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.928328 715 raft_consensus.cc:705] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:28.928516 715 consensus_queue.cc:229] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.928750 437 heartbeater.cc:446] Master 127.0.58.190:37585 was elected leader, sending a full tablet report... I0720 00:42:28.930701 716 ts_tablet_manager.cc:1087] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.075s user 0.003s sys 0.004s I0720 00:42:28.930833 714 ts_tablet_manager.cc:1087] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.072s user 0.007s sys 0.004s I0720 00:42:28.931164 721 ts_tablet_manager.cc:1087] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.058s user 0.005s sys 0.002s I0720 00:42:28.931890 715 ts_tablet_manager.cc:1087] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.079s user 0.009s sys 0.001s I0720 00:42:28.931579 722 raft_consensus.cc:340] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.932103 722 raft_consensus.cc:366] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.931613 720 raft_consensus.cc:340] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.932296 720 raft_consensus.cc:366] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.932349 717 ts_tablet_manager.cc:1087] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.069s user 0.003s sys 0.003s I0720 00:42:28.932351 720 raft_consensus.cc:705] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Initialized, Role: FOLLOWER I0720 00:42:28.932631 718 raft_consensus.cc:340] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.932838 718 raft_consensus.cc:366] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.932883 718 raft_consensus.cc:705] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Initialized, Role: FOLLOWER I0720 00:42:28.933018 720 consensus_queue.cc:229] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.932184 722 raft_consensus.cc:705] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Initialized, Role: FOLLOWER I0720 00:42:28.933369 723 raft_consensus.cc:340] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.933563 723 raft_consensus.cc:366] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.933610 723 raft_consensus.cc:705] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Initialized, Role: FOLLOWER I0720 00:42:28.933559 722 consensus_queue.cc:229] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.933778 723 consensus_queue.cc:229] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.931579 719 raft_consensus.cc:340] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:28.933063 718 consensus_queue.cc:229] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.934666 564 heartbeater.cc:446] Master 127.0.58.190:37585 was elected leader, sending a full tablet report... I0720 00:42:28.934355 719 raft_consensus.cc:366] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.935647 719 raft_consensus.cc:705] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Initialized, Role: FOLLOWER I0720 00:42:28.935859 719 consensus_queue.cc:229] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:28.936236 729 raft_consensus.cc:340] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.936573 729 raft_consensus.cc:366] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.936659 729 raft_consensus.cc:705] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Initialized, Role: FOLLOWER I0720 00:42:28.937165 737 raft_consensus.cc:340] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.937371 737 raft_consensus.cc:366] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.937423 737 raft_consensus.cc:705] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Initialized, Role: FOLLOWER I0720 00:42:28.937981 730 raft_consensus.cc:340] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.939198 730 raft_consensus.cc:366] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.939247 730 raft_consensus.cc:705] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Initialized, Role: FOLLOWER I0720 00:42:28.937677 727 raft_consensus.cc:340] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.939471 727 raft_consensus.cc:366] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.939520 727 raft_consensus.cc:705] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Initialized, Role: FOLLOWER I0720 00:42:28.939929 727 consensus_queue.cc:229] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.940490 734 raft_consensus.cc:340] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:28.940635 734 raft_consensus.cc:366] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:28.940681 734 raft_consensus.cc:705] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Initialized, Role: FOLLOWER I0720 00:42:28.940831 734 consensus_queue.cc:229] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:28.941218 737 consensus_queue.cc:229] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.941747 690 heartbeater.cc:446] Master 127.0.58.190:37585 was elected leader, sending a full tablet report... I0720 00:42:28.942517 737 ts_tablet_manager.cc:1087] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent starting tablet: real 0.045s user 0.002s sys 0.005s I0720 00:42:28.942764 734 ts_tablet_manager.cc:1087] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent starting tablet: real 0.036s user 0.001s sys 0.006s I0720 00:42:28.942958 730 consensus_queue.cc:229] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:28.943342 730 ts_tablet_manager.cc:1087] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent starting tablet: real 0.046s user 0.000s sys 0.009s I0720 00:42:28.943563 729 consensus_queue.cc:229] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } W0720 00:42:28.944797 691 tablet.cc:1799] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:28.945621 729 ts_tablet_manager.cc:1087] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent starting tablet: real 0.054s user 0.011s sys 0.000s I0720 00:42:28.950706 720 ts_tablet_manager.cc:1087] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: Time spent starting tablet: real 0.073s user 0.005s sys 0.005s I0720 00:42:28.951063 723 ts_tablet_manager.cc:1087] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c: Time spent starting tablet: real 0.074s user 0.001s sys 0.011s I0720 00:42:28.951865 719 ts_tablet_manager.cc:1087] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c: Time spent starting tablet: real 0.076s user 0.025s sys 0.030s I0720 00:42:28.953277 722 ts_tablet_manager.cc:1087] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c: Time spent starting tablet: real 0.077s user 0.021s sys 0.025s I0720 00:42:28.953513 718 ts_tablet_manager.cc:1087] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: Time spent starting tablet: real 0.083s user 0.009s sys 0.001s I0720 00:42:28.953727 727 ts_tablet_manager.cc:1087] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc: Time spent starting tablet: real 0.053s user 0.012s sys 0.004s I0720 00:42:29.019449 753 raft_consensus.cc:472] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:29.019702 753 raft_consensus.cc:494] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:29.021055 753 leader_election.cc:296] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) I0720 00:42:29.024554 749 raft_consensus.cc:472] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:29.024724 749 raft_consensus.cc:494] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.026026 749 leader_election.cc:296] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.028826 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7cab46396b7e471f89b67a9bee891c6a" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" is_pre_election: true I0720 00:42:29.029184 521 raft_consensus.cc:2350] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 0. I0720 00:42:29.029820 580 leader_election.cc:310] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfcaafb29b4431b9ac0f78f5dec9cc, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:29.030333 753 raft_consensus.cc:2644] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:29.030448 753 raft_consensus.cc:472] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:29.030509 753 raft_consensus.cc:2890] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.031976 749 raft_consensus.cc:472] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:29.032080 749 raft_consensus.cc:494] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.032487 749 leader_election.cc:296] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.033088 753 raft_consensus.cc:494] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:29.033656 753 leader_election.cc:296] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 1 election: Requested vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) I0720 00:42:29.036231 744 raft_consensus.cc:472] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:29.036450 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7cab46396b7e471f89b67a9bee891c6a" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" I0720 00:42:29.036594 521 raft_consensus.cc:2890] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.036490 744 raft_consensus.cc:494] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.037621 394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7cab46396b7e471f89b67a9bee891c6a" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" is_pre_election: true I0720 00:42:29.037930 393 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7cab46396b7e471f89b67a9bee891c6a" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" I0720 00:42:29.038102 394 raft_consensus.cc:2350] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 0. I0720 00:42:29.038280 393 raft_consensus.cc:2325] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 25bfcaafb29b4431b9ac0f78f5dec9cc for term 1 because replica is already servicing an update from a current leader or another vote. I0720 00:42:29.039089 521 raft_consensus.cc:2350] T 7cab46396b7e471f89b67a9bee891c6a P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 1. I0720 00:42:29.039448 580 leader_election.cc:310] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25bfcaafb29b4431b9ac0f78f5dec9cc, 4deaf10768b24896bcd716f79c982b0c; no voters: 0e2d29c89c754e8ea12ed6730aa4c85d I0720 00:42:29.039729 753 raft_consensus.cc:2644] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:29.040199 753 raft_consensus.cc:667] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 LEADER]: Becoming Leader. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Running, Role: LEADER I0720 00:42:29.042106 744 leader_election.cc:296] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.042393 753 consensus_queue.cc:206] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:29.045778 394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8668997e728c430da205e7d36533f74a" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" is_pre_election: true I0720 00:42:29.045732 393 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6da351b3ad134fc78c052d34aafdcbc0" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" is_pre_election: true I0720 00:42:29.045922 394 raft_consensus.cc:2350] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 0. I0720 00:42:29.045920 393 raft_consensus.cc:2350] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 0. I0720 00:42:29.046347 454 leader_election.cc:310] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:29.046748 749 raft_consensus.cc:2644] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:29.046764 454 leader_election.cc:310] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:29.046823 749 raft_consensus.cc:472] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:29.046893 749 raft_consensus.cc:2890] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.047065 756 raft_consensus.cc:2644] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:29.047137 756 raft_consensus.cc:472] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:29.047226 756 raft_consensus.cc:2890] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.046680 270 catalog_manager.cc:4067] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: term changed from 0 to 1, leader changed from to 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131). New cstate: current_term: 1 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:29.050410 749 raft_consensus.cc:494] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.050320 756 raft_consensus.cc:494] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.051075 756 leader_election.cc:296] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 election: Requested vote from peers 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.051345 393 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6da351b3ad134fc78c052d34aafdcbc0" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" I0720 00:42:29.051491 393 raft_consensus.cc:2890] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.051651 749 leader_election.cc:296] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 election: Requested vote from peers 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.052183 394 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8668997e728c430da205e7d36533f74a" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" I0720 00:42:29.052330 394 raft_consensus.cc:2890] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.052872 647 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6da351b3ad134fc78c052d34aafdcbc0" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" is_pre_election: true I0720 00:42:29.053185 646 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8668997e728c430da205e7d36533f74a" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" is_pre_election: true I0720 00:42:29.054116 647 raft_consensus.cc:2350] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 0. I0720 00:42:29.054587 393 raft_consensus.cc:2350] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 1. I0720 00:42:29.054342 645 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6da351b3ad134fc78c052d34aafdcbc0" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" I0720 00:42:29.054162 644 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8668997e728c430da205e7d36533f74a" candidate_uuid: "4deaf10768b24896bcd716f79c982b0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" I0720 00:42:29.054721 645 raft_consensus.cc:2890] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.054813 646 raft_consensus.cc:2350] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 0. I0720 00:42:29.055016 454 leader_election.cc:310] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:29.055181 644 raft_consensus.cc:2325] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Leader election vote request: Denying vote to candidate 4deaf10768b24896bcd716f79c982b0c for term 1 because replica is already servicing an update from a current leader or another vote. I0720 00:42:29.055344 749 raft_consensus.cc:2644] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:29.056913 749 raft_consensus.cc:667] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Becoming Leader. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Running, Role: LEADER I0720 00:42:29.057242 394 raft_consensus.cc:2350] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 1. I0720 00:42:29.057274 749 consensus_queue.cc:206] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.057613 454 leader_election.cc:310] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 4deaf10768b24896bcd716f79c982b0c; no voters: 25bfcaafb29b4431b9ac0f78f5dec9cc I0720 00:42:29.057888 645 raft_consensus.cc:2350] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4deaf10768b24896bcd716f79c982b0c in term 1. I0720 00:42:29.057977 756 raft_consensus.cc:2644] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:29.058125 756 raft_consensus.cc:667] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Becoming Leader. State: Replica: 4deaf10768b24896bcd716f79c982b0c, State: Running, Role: LEADER I0720 00:42:29.058348 756 consensus_queue.cc:206] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.061357 645 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" is_pre_election: true I0720 00:42:29.061552 645 raft_consensus.cc:2350] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 0. I0720 00:42:29.061952 325 leader_election.cc:310] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 25bfcaafb29b4431b9ac0f78f5dec9cc; no voters: I0720 00:42:29.062471 744 raft_consensus.cc:2644] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:29.062577 744 raft_consensus.cc:472] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:29.062676 744 raft_consensus.cc:2890] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.062786 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" is_pre_election: true I0720 00:42:29.062970 521 raft_consensus.cc:2350] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 0. I0720 00:42:29.063792 270 catalog_manager.cc:4067] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c reported cstate change: term changed from 0 to 1, leader changed from to 4deaf10768b24896bcd716f79c982b0c (127.0.58.130). New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:29.064620 744 raft_consensus.cc:494] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.065901 744 leader_election.cc:296] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 election: Requested vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.066234 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" I0720 00:42:29.066306 645 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" I0720 00:42:29.066402 521 raft_consensus.cc:2890] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.066448 645 raft_consensus.cc:2890] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.068150 645 raft_consensus.cc:2350] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 1. I0720 00:42:29.068224 521 raft_consensus.cc:2350] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 1. I0720 00:42:29.068442 325 leader_election.cc:310] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 25bfcaafb29b4431b9ac0f78f5dec9cc; no voters: I0720 00:42:29.068226 270 catalog_manager.cc:4067] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c reported cstate change: term changed from 0 to 1, leader changed from to 4deaf10768b24896bcd716f79c982b0c (127.0.58.130). New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:29.068696 744 raft_consensus.cc:2644] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:29.068810 744 raft_consensus.cc:667] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 LEADER]: Becoming Leader. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Running, Role: LEADER I0720 00:42:29.069196 744 consensus_queue.cc:206] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.072125 270 catalog_manager.cc:4067] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d reported cstate change: term changed from 0 to 1, leader changed from to 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129). New cstate: current_term: 1 leader_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } W0720 00:42:29.167443 438 tablet.cc:1799] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:42:29.178328 565 tablet.cc:1799] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:29.280678 744 raft_consensus.cc:472] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:29.280867 744 raft_consensus.cc:494] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.281461 744 leader_election.cc:296] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.282150 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" is_pre_election: true I0720 00:42:29.282145 645 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" is_pre_election: true I0720 00:42:29.282333 521 raft_consensus.cc:2350] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 0. I0720 00:42:29.282352 645 raft_consensus.cc:2350] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 0. I0720 00:42:29.282728 327 leader_election.cc:310] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:29.283010 744 raft_consensus.cc:2644] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:29.283062 744 raft_consensus.cc:472] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:29.283114 744 raft_consensus.cc:2890] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.285459 744 raft_consensus.cc:494] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.285979 744 leader_election.cc:296] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 election: Requested vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:29.286447 645 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" I0720 00:42:29.286447 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" I0720 00:42:29.286605 521 raft_consensus.cc:2890] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.286607 645 raft_consensus.cc:2890] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.288630 645 raft_consensus.cc:2350] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 1. I0720 00:42:29.288630 521 raft_consensus.cc:2350] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e2d29c89c754e8ea12ed6730aa4c85d in term 1. I0720 00:42:29.288998 325 leader_election.cc:310] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e2d29c89c754e8ea12ed6730aa4c85d, 25bfcaafb29b4431b9ac0f78f5dec9cc; no voters: I0720 00:42:29.289279 744 raft_consensus.cc:2644] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:29.289404 744 raft_consensus.cc:667] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 LEADER]: Becoming Leader. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Running, Role: LEADER I0720 00:42:29.289604 744 consensus_queue.cc:206] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.291800 270 catalog_manager.cc:4067] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d reported cstate change: term changed from 0 to 1, leader changed from to 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129). New cstate: current_term: 1 leader_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:29.386929 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 304 I0720 00:42:29.387575 434 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:29.445452 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 304 W0720 00:42:29.456432 580 connection.cc:655] client connection to 127.0.58.129:34255 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0720 00:42:29.456605 580 consensus_peers.cc:458] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:29.457223 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.129:34255 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_port=45131 --webserver_interface=127.0.58.129 --tserver_master_addrs=127.0.58.190:37585 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** --env_inject_eio=1.0 --crash_on_eio=false --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:29.504068 753 consensus_queue.cc:984] T 7cab46396b7e471f89b67a9bee891c6a P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:29.522689 749 consensus_queue.cc:984] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0720 00:42:29.527035 454 consensus_peers.cc:458] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:29.604751 775 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:29.605000 775 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:29.607070 775 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:29.607136 775 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:29.607234 775 flags.cc:404] Enabled experimental flag: --crash_on_eio=false W0720 00:42:29.607301 775 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:42:29.611104 775 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:34255 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=45131 --tserver_master_addrs=127.0.58.190:37585 --env_inject_eio=1 --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:29.618849 775 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 36016us W0720 00:42:29.619169 775 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:42:29.619243 775 fault_injection.cc:45] THIS SERVER MAY CRASH! I0720 00:42:29.619833 775 fs_manager.cc:263] Metadata directory not provided I0720 00:42:29.619921 775 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory E0720 00:42:29.621313 775 env_posix.cc:316] I/O error, context: INJECTED FAILURE E0720 00:42:29.623162 775 env_posix.cc:316] I/O error, context: INJECTED FAILURE I0720 00:42:29.623252 775 block_manager_util.cc:139] Instance is unhealthy: IO error: Failed to read metadata file from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/data/block_manager_instance: INJECTED FAILURE: Input/output error (error 5) I0720 00:42:29.629070 775 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.003s sys 0.002s I0720 00:42:29.630105 775 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:29.630652 775 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:29.631290 775 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0720 00:42:29.631368 775 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" format_stamp: "Formatted at 2019-07-20 00:42:28 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:29.632233 775 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal 4 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:29.650218 775 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:29.651964 775 ts_tablet_manager.cc:345] Loading tablet metadata (0/5 complete) I0720 00:42:29.658416 645 raft_consensus.cc:2890] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.661345 749 consensus_queue.cc:984] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:29.663117 775 ts_tablet_manager.cc:357] Loaded tablet metadata (5 total tablets, 5 live tablets) I0720 00:42:29.663187 775 ts_tablet_manager.cc:363] Registering tablets (0/5 complete) W0720 00:42:29.663354 454 consensus_peers.cc:458] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:29.667657 798 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. E0720 00:42:29.668041 798 ts_tablet_manager.cc:1080] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:29.668206 798 ts_tablet_manager.cc:1059] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0720 00:42:29.668344 798 tablet_replica.cc:273] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: stopping tablet replica I0720 00:42:29.668488 798 raft_consensus.cc:2147] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:29.669858 798 raft_consensus.cc:2174] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:29.670312 799 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. E0720 00:42:29.670464 799 ts_tablet_manager.cc:1080] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:29.670578 799 ts_tablet_manager.cc:1059] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:42:29.670634 799 tablet_replica.cc:273] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: stopping tablet replica I0720 00:42:29.670712 799 raft_consensus.cc:2147] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:29.670791 799 raft_consensus.cc:2174] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:29.672646 798 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:29.677067 799 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:29.678967 775 ts_tablet_manager.cc:377] Registered 5 tablets I0720 00:42:29.678987 800 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. E0720 00:42:29.679301 800 ts_tablet_manager.cc:1080] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:29.680574 800 ts_tablet_manager.cc:1059] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0720 00:42:29.680670 800 tablet_replica.cc:273] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: stopping tablet replica I0720 00:42:29.680927 800 raft_consensus.cc:2147] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:29.681077 800 raft_consensus.cc:2174] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:29.689504 799 log.cc:525] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Log is configured to *not* fsync() on all Append() calls I0720 00:42:29.694263 799 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:29.695056 799 tablet_bootstrap.cc:439] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap complete. I0720 00:42:29.696632 798 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0720 00:42:29.696743 799 ts_tablet_manager.cc:1059] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.009s I0720 00:42:29.697428 798 tablet_bootstrap.cc:439] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap complete. I0720 00:42:29.698478 798 ts_tablet_manager.cc:1059] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.010s I0720 00:42:29.713125 798 raft_consensus.cc:340] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.713413 799 raft_consensus.cc:340] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:29.715987 799 raft_consensus.cc:366] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:29.716074 799 raft_consensus.cc:705] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:29.716256 798 raft_consensus.cc:705] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:29.716789 799 consensus_queue.cc:229] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:29.716812 798 consensus_queue.cc:229] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:29.718767 798 ts_tablet_manager.cc:1087] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.020s user 0.006s sys 0.010s I0720 00:42:29.719261 799 ts_tablet_manager.cc:1087] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.022s user 0.011s sys 0.009s I0720 00:42:29.726802 775 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:34255 I0720 00:42:29.727838 775 webserver.cc:374] Webserver started at http://127.0.58.129:45131/ using document root and password file I0720 00:42:29.728516 775 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb W0720 00:42:29.730065 910 tablet.cc:1799] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:29.738242 909 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37585 I0720 00:42:29.738320 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 775 I0720 00:42:29.738451 909 heartbeater.cc:405] Registering TS with master... I0720 00:42:29.738919 909 heartbeater.cc:454] Master 127.0.58.190:37585 requested a full tablet report, sending... I0720 00:42:29.741019 268 ts_manager.cc:161] Re-registered known tserver with Master: 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) W0720 00:42:29.741403 268 catalog_manager.cc:3988] Tablet 8668997e728c430da205e7d36533f74a (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) has failed on TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): IO error: some tablet data is in a failed directory W0720 00:42:29.741503 268 catalog_manager.cc:3988] Tablet 6da351b3ad134fc78c052d34aafdcbc0 (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) has failed on TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): IO error: some tablet data is in a failed directory W0720 00:42:29.741636 268 catalog_manager.cc:3988] Tablet 34dd0711864545e1beb62517f20f557e (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) has failed on TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): IO error: some tablet data is in a failed directory I0720 00:42:29.742652 268 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:36539 Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 7cab46396b7e471f89b67a9bee891c6a of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING [LEADER] 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): RUNNING All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | A B C | 0 | -1 | Yes Tablet ea63d074a81e47c7910a7097efab544e of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): RUNNING [LEADER] 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet 34dd0711864545e1beb62517f20f557e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 0e2d29c89c754e8ea12ed6730aa4c85d C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | [config not available] | | | C | A B C | 1 | -1 | Yes Tablet 8668997e728c430da205e7d36533f74a of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 6da351b3ad134fc78c052d34aafdcbc0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 0 | 0 | 3 | 2 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:29.834851 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:29.909862 864 raft_consensus.cc:2890] T 7cab46396b7e471f89b67a9bee891c6a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:29.913094 909 heartbeater.cc:446] Master 127.0.58.190:37585 was elected leader, sending a full tablet report... W0720 00:42:29.916412 268 catalog_manager.cc:3988] Tablet 8668997e728c430da205e7d36533f74a (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) has failed on TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): IO error: some tablet data is in a failed directory W0720 00:42:29.916532 268 catalog_manager.cc:3988] Tablet 6da351b3ad134fc78c052d34aafdcbc0 (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) has failed on TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): IO error: some tablet data is in a failed directory W0720 00:42:29.916674 268 catalog_manager.cc:3988] Tablet 34dd0711864545e1beb62517f20f557e (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) has failed on TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): IO error: some tablet data is in a failed directory I0720 00:42:30.039969 749 raft_consensus.cc:922] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: Attempting to remove follower 0e2d29c89c754e8ea12ed6730aa4c85d from the Raft config. Reason: The tablet replica hosted on peer 0e2d29c89c754e8ea12ed6730aa4c85d has failed I0720 00:42:30.040899 749 consensus_queue.cc:206] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:30.042429 645 raft_consensus.cc:1184] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Refusing update from remote peer 4deaf10768b24896bcd716f79c982b0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:30.045326 756 consensus_queue.cc:984] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:30.047360 749 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.047873 645 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.051795 268 catalog_manager.cc:4067] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c reported cstate change: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet ea63d074a81e47c7910a7097efab544e of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): RUNNING [LEADER] 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet 34dd0711864545e1beb62517f20f557e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 0e2d29c89c754e8ea12ed6730aa4c85d C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | [config not available] | | | C | A B C | 1 | -1 | Yes Tablet 8668997e728c430da205e7d36533f74a of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 6da351b3ad134fc78c052d34aafdcbc0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | B* C | 1 | 2 | No C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 1 | 0 | 3 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:30.058468 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:30.060073 268 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6da351b3ad134fc78c052d34aafdcbc0 on 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 2) I0720 00:42:30.060370 268 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet 6da351b3ad134fc78c052d34aafdcbc0 (attempt 1) I0720 00:42:30.062494 521 consensus_queue.cc:206] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:30.064158 645 raft_consensus.cc:1184] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Refusing update from remote peer 4deaf10768b24896bcd716f79c982b0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) W0720 00:42:30.064471 454 consensus_peers.cc:458] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:30.064597 756 consensus_queue.cc:984] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:42:30.066431 749 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.066859 844 tablet_service.cc:1066] Processing DeleteTablet for tablet 6da351b3ad134fc78c052d34aafdcbc0 with delete_type TABLET_DATA_TOMBSTONED (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:37654 I0720 00:42:30.066929 645 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.070336 268 catalog_manager.cc:4067] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.071735 253 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6da351b3ad134fc78c052d34aafdcbc0 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:30.072455 969 ts_tablet_manager.cc:1400] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:30.076592 969 ts_tablet_manager.cc:1413] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:30.076689 969 log.cc:990] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal/wals/6da351b3ad134fc78c052d34aafdcbc0 I0720 00:42:30.077373 253 catalog_manager.cc:3437] TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): tablet 6da351b3ad134fc78c052d34aafdcbc0 (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) successfully deleted I0720 00:42:30.138000 756 raft_consensus.cc:922] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: Attempting to remove follower 0e2d29c89c754e8ea12ed6730aa4c85d from the Raft config. Reason: The tablet replica hosted on peer 0e2d29c89c754e8ea12ed6730aa4c85d has failed I0720 00:42:30.138522 756 consensus_queue.cc:206] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:30.140027 645 raft_consensus.cc:1184] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Refusing update from remote peer 4deaf10768b24896bcd716f79c982b0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:30.140489 749 consensus_queue.cc:984] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:30.142287 756 raft_consensus.cc:2792] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.142737 645 raft_consensus.cc:2792] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.145622 269 catalog_manager.cc:4067] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c reported cstate change: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.148448 269 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 8668997e728c430da205e7d36533f74a on 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 2) I0720 00:42:30.148644 269 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet 8668997e728c430da205e7d36533f74a (attempt 1) I0720 00:42:30.149206 844 tablet_service.cc:1066] Processing DeleteTablet for tablet 8668997e728c430da205e7d36533f74a with delete_type TABLET_DATA_TOMBSTONED (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:37654 I0720 00:42:30.149510 969 ts_tablet_manager.cc:1400] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:30.149929 521 consensus_queue.cc:206] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:30.151312 645 raft_consensus.cc:1184] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Refusing update from remote peer 4deaf10768b24896bcd716f79c982b0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) W0720 00:42:30.151746 454 consensus_peers.cc:458] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:30.151829 749 consensus_queue.cc:984] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:42:30.153779 969 ts_tablet_manager.cc:1413] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:30.153862 969 log.cc:990] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal/wals/8668997e728c430da205e7d36533f74a I0720 00:42:30.153620 756 raft_consensus.cc:2792] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.154187 645 raft_consensus.cc:2792] T 8668997e728c430da205e7d36533f74a P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.154629 253 catalog_manager.cc:3437] TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): tablet 8668997e728c430da205e7d36533f74a (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) successfully deleted I0720 00:42:30.155853 253 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet 8668997e728c430da205e7d36533f74a with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:30.157250 269 catalog_manager.cc:4067] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c reported cstate change: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet ea63d074a81e47c7910a7097efab544e of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): RUNNING [LEADER] 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet 34dd0711864545e1beb62517f20f557e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 0e2d29c89c754e8ea12ed6730aa4c85d C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | [config not available] | | | C | A B C | 1 | -1 | Yes Tablet 8668997e728c430da205e7d36533f74a of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_TOMBSTONED Last status: Deleted tablet blocks from disk All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 3 | Yes B | A* B C | 1 | 3 | Yes C | [config not available] | | | Tablet 6da351b3ad134fc78c052d34aafdcbc0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_TOMBSTONED Last status: Deleted tablet blocks from disk All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 3 | Yes B | A* B C | 1 | 3 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 1 | 0 | 3 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:30.357908 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:30.665370 756 raft_consensus.cc:922] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: Attempting to remove follower 0e2d29c89c754e8ea12ed6730aa4c85d from the Raft config. Reason: The tablet replica hosted on peer 0e2d29c89c754e8ea12ed6730aa4c85d has failed I0720 00:42:30.665848 756 consensus_queue.cc:206] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:30.667306 645 raft_consensus.cc:1184] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Refusing update from remote peer 4deaf10768b24896bcd716f79c982b0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:30.667824 993 consensus_queue.cc:984] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:30.669564 756 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.670156 645 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.672993 268 catalog_manager.cc:4067] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c reported cstate change: config changed from index 3 to 4, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:30.682984 268 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6da351b3ad134fc78c052d34aafdcbc0 on 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 4) I0720 00:42:30.683190 268 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet 6da351b3ad134fc78c052d34aafdcbc0 (attempt 1) I0720 00:42:30.683740 844 tablet_service.cc:1066] Processing DeleteTablet for tablet 6da351b3ad134fc78c052d34aafdcbc0 with delete_type TABLET_DATA_TOMBSTONED (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37654 I0720 00:42:30.684429 521 consensus_queue.cc:206] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:30.685955 645 raft_consensus.cc:1184] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Refusing update from remote peer 4deaf10768b24896bcd716f79c982b0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) W0720 00:42:30.686408 454 consensus_peers.cc:458] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:30.686425 993 consensus_queue.cc:984] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0720 00:42:30.688163 756 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.688690 645 raft_consensus.cc:2792] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.690198 253 catalog_manager.cc:3437] TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): tablet 6da351b3ad134fc78c052d34aafdcbc0 (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) successfully deleted I0720 00:42:30.690897 253 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet 6da351b3ad134fc78c052d34aafdcbc0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:30.692428 268 catalog_manager.cc:4067] T 6da351b3ad134fc78c052d34aafdcbc0 P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: config changed from index 4 to 5, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "4deaf10768b24896bcd716f79c982b0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:30.715390 755 raft_consensus.cc:472] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:30.715538 755 raft_consensus.cc:494] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:30.716086 755 leader_election.cc:296] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255), 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913) I0720 00:42:30.716668 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" is_pre_election: true I0720 00:42:30.716851 521 raft_consensus.cc:2350] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 1. I0720 00:42:30.716828 863 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" is_pre_election: true I0720 00:42:30.717227 863 raft_consensus.cc:2297] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 25bfcaafb29b4431b9ac0f78f5dec9cc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0720 00:42:30.717290 580 leader_election.cc:310] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfcaafb29b4431b9ac0f78f5dec9cc, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:30.717670 755 raft_consensus.cc:2644] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:42:30.717737 755 raft_consensus.cc:472] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:30.717792 755 raft_consensus.cc:2890] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:30.719943 755 raft_consensus.cc:494] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:30.720418 755 leader_election.cc:296] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 election: Requested vote from peers 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255), 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913) I0720 00:42:30.720885 863 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" I0720 00:42:30.720919 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ea63d074a81e47c7910a7097efab544e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" I0720 00:42:30.721021 863 raft_consensus.cc:2890] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:30.721073 521 raft_consensus.cc:2890] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:30.722955 521 raft_consensus.cc:2350] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 2. I0720 00:42:30.723143 863 raft_consensus.cc:2297] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 25bfcaafb29b4431b9ac0f78f5dec9cc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0720 00:42:30.723278 580 leader_election.cc:310] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25bfcaafb29b4431b9ac0f78f5dec9cc, 4deaf10768b24896bcd716f79c982b0c; no voters: I0720 00:42:30.723553 755 raft_consensus.cc:2644] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:42:30.724758 755 raft_consensus.cc:667] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 LEADER]: Becoming Leader. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Running, Role: LEADER I0720 00:42:30.725008 755 consensus_queue.cc:206] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:30.727161 269 catalog_manager.cc:4067] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: term changed from 1 to 2, leader changed from 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) to 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131). New cstate: current_term: 2 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet ea63d074a81e47c7910a7097efab544e of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING [LEADER] All reported replicas are: A = 0e2d29c89c754e8ea12ed6730aa4c85d B = 4deaf10768b24896bcd716f79c982b0c C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C | 2 | -1 | Yes C | A B C* | 2 | -1 | Yes Tablet 34dd0711864545e1beb62517f20f557e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 0e2d29c89c754e8ea12ed6730aa4c85d C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | [config not available] | | | C | A B C | 1 | -1 | Yes Tablet 8668997e728c430da205e7d36533f74a of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_TOMBSTONED Last status: Deleted tablet blocks from disk All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 3 | Yes B | A* B C | 1 | 3 | Yes C | [config not available] | | | Tablet 6da351b3ad134fc78c052d34aafdcbc0 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_TOMBSTONED Last status: Deleted tablet blocks from disk All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 5 | Yes B | A* B C | 1 | 5 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 1 | 0 | 3 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:30.759881 1023 ts_tablet_manager.cc:682] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Initiating tablet copy from peer 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913) I0720 00:42:30.760109 1023 tablet_copy_client.cc:204] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:30.760288 1023 tablet_copy_client.cc:241] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.130:38913 I0720 00:42:30.763151 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:30.766063 541 tablet_copy_service.cc:135] P 4deaf10768b24896bcd716f79c982b0c: Received BeginTabletCopySession request for tablet 8668997e728c430da205e7d36533f74a from peer 0e2d29c89c754e8ea12ed6730aa4c85d ({username='slave'} at 127.0.58.129:41639) I0720 00:42:30.766297 541 tablet_copy_service.cc:156] P 4deaf10768b24896bcd716f79c982b0c: Beginning new tablet copy session on tablet 8668997e728c430da205e7d36533f74a from peer 0e2d29c89c754e8ea12ed6730aa4c85d at {username='slave'} at 127.0.58.129:41639: session id = 0e2d29c89c754e8ea12ed6730aa4c85d-8668997e728c430da205e7d36533f74a I0720 00:42:30.768324 541 tablet_copy_source_session.cc:202] T 8668997e728c430da205e7d36533f74a P 4deaf10768b24896bcd716f79c982b0c: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:30.769603 1023 ts_tablet_manager.cc:1400] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:42:30.773955 1023 ts_tablet_manager.cc:1413] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:30.776180 1023 tablet_copy_client.cc:579] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Starting download of 0 data blocks... I0720 00:42:30.776386 1023 tablet_copy_client.cc:542] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Starting download of 1 WAL segments... I0720 00:42:30.777961 1023 tablet_copy_client.cc:418] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:30.779928 1023 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:30.785169 1023 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:30.785521 1023 tablet_bootstrap.cc:439] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap complete. I0720 00:42:30.785670 1023 ts_tablet_manager.cc:1059] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s I0720 00:42:30.786975 1023 raft_consensus.cc:340] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:30.787129 1023 raft_consensus.cc:705] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:30.787308 1023 consensus_queue.cc:229] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:30.788273 1023 ts_tablet_manager.cc:1087] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s I0720 00:42:30.788965 541 tablet_copy_service.cc:325] P 4deaf10768b24896bcd716f79c982b0c: Request end of tablet copy session 0e2d29c89c754e8ea12ed6730aa4c85d-8668997e728c430da205e7d36533f74a received from {username='slave'} at 127.0.58.129:41639 I0720 00:42:30.789080 541 tablet_copy_service.cc:413] P 4deaf10768b24896bcd716f79c982b0c: ending tablet copy session 0e2d29c89c754e8ea12ed6730aa4c85d-8668997e728c430da205e7d36533f74a on tablet 8668997e728c430da205e7d36533f74a with peer 0e2d29c89c754e8ea12ed6730aa4c85d I0720 00:42:31.094799 755 raft_consensus.cc:472] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:31.094992 755 raft_consensus.cc:494] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:31.095566 755 leader_election.cc:296] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) I0720 00:42:31.096212 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" is_pre_election: true I0720 00:42:31.096231 863 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" is_pre_election: true I0720 00:42:31.096421 521 raft_consensus.cc:2350] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 1. W0720 00:42:31.096812 580 leader_election.cc:348] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Illegal state: cannot vote while shut down I0720 00:42:31.097007 580 leader_election.cc:310] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25bfcaafb29b4431b9ac0f78f5dec9cc, 4deaf10768b24896bcd716f79c982b0c; no voters: 0e2d29c89c754e8ea12ed6730aa4c85d I0720 00:42:31.097247 755 raft_consensus.cc:2644] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:42:31.097297 755 raft_consensus.cc:472] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:31.097357 755 raft_consensus.cc:2890] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:31.098948 755 raft_consensus.cc:494] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:31.099412 755 leader_election.cc:296] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 election: Requested vote from peers 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913), 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) I0720 00:42:31.099869 521 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4deaf10768b24896bcd716f79c982b0c" I0720 00:42:31.099939 863 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "34dd0711864545e1beb62517f20f557e" candidate_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" I0720 00:42:31.100039 521 raft_consensus.cc:2890] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 1 FOLLOWER]: Advancing to term 2 W0720 00:42:31.100392 580 leader_election.cc:348] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Illegal state: cannot vote while shut down I0720 00:42:31.101673 521 raft_consensus.cc:2350] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25bfcaafb29b4431b9ac0f78f5dec9cc in term 2. I0720 00:42:31.101992 580 leader_election.cc:310] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25bfcaafb29b4431b9ac0f78f5dec9cc, 4deaf10768b24896bcd716f79c982b0c; no voters: 0e2d29c89c754e8ea12ed6730aa4c85d I0720 00:42:31.102250 755 raft_consensus.cc:2644] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:42:31.102366 755 raft_consensus.cc:667] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 LEADER]: Becoming Leader. State: Replica: 25bfcaafb29b4431b9ac0f78f5dec9cc, State: Running, Role: LEADER I0720 00:42:31.102556 755 consensus_queue.cc:206] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:31.104665 268 catalog_manager.cc:4067] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: term changed from 1 to 2, leader changed from 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) to 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131). New cstate: current_term: 2 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:31.148536 863 raft_consensus.cc:1184] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch) I0720 00:42:31.148654 863 pending_rounds.cc:78] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Aborting all transactions after (but not including) 0 I0720 00:42:31.148684 863 pending_rounds.cc:99] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d: Aborting uncommitted NO_OP operation due to leader change: 1.1 I0720 00:42:31.148764 863 raft_consensus.cc:2727] T ea63d074a81e47c7910a7097efab544e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0720 00:42:31.149410 755 consensus_queue.cc:984] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:31.158840 521 raft_consensus.cc:1184] T ea63d074a81e47c7910a7097efab544e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0720 00:42:31.159313 755 consensus_queue.cc:984] T ea63d074a81e47c7910a7097efab544e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:31.184460 863 raft_consensus.cc:1126] T 8668997e728c430da205e7d36533f74a P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[] I0720 00:42:31.284997 1023 ts_tablet_manager.cc:682] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Initiating tablet copy from peer 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913) I0720 00:42:31.285095 1023 tablet_copy_client.cc:204] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:31.285243 1023 tablet_copy_client.cc:241] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.130:38913 I0720 00:42:31.285917 541 tablet_copy_service.cc:135] P 4deaf10768b24896bcd716f79c982b0c: Received BeginTabletCopySession request for tablet 6da351b3ad134fc78c052d34aafdcbc0 from peer 0e2d29c89c754e8ea12ed6730aa4c85d ({username='slave'} at 127.0.58.129:41639) I0720 00:42:31.286098 541 tablet_copy_service.cc:156] P 4deaf10768b24896bcd716f79c982b0c: Beginning new tablet copy session on tablet 6da351b3ad134fc78c052d34aafdcbc0 from peer 0e2d29c89c754e8ea12ed6730aa4c85d at {username='slave'} at 127.0.58.129:41639: session id = 0e2d29c89c754e8ea12ed6730aa4c85d-6da351b3ad134fc78c052d34aafdcbc0 I0720 00:42:31.287803 541 tablet_copy_source_session.cc:202] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:31.288787 1023 ts_tablet_manager.cc:1400] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:42:31.293143 1023 ts_tablet_manager.cc:1413] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:31.295859 1023 tablet_copy_client.cc:579] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Starting download of 0 data blocks... I0720 00:42:31.296054 1023 tablet_copy_client.cc:542] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Starting download of 1 WAL segments... I0720 00:42:31.297432 1023 tablet_copy_client.cc:418] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:31.299770 1023 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:31.307673 1023 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:31.308085 1023 tablet_bootstrap.cc:439] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap complete. I0720 00:42:31.308276 1023 ts_tablet_manager.cc:1059] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s I0720 00:42:31.308981 1023 raft_consensus.cc:340] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:31.309149 1023 raft_consensus.cc:705] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:31.309322 1023 consensus_queue.cc:229] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:31.309743 1023 ts_tablet_manager.cc:1087] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0720 00:42:31.310384 541 tablet_copy_service.cc:325] P 4deaf10768b24896bcd716f79c982b0c: Request end of tablet copy session 0e2d29c89c754e8ea12ed6730aa4c85d-6da351b3ad134fc78c052d34aafdcbc0 received from {username='slave'} at 127.0.58.129:41639 I0720 00:42:31.310473 541 tablet_copy_service.cc:413] P 4deaf10768b24896bcd716f79c982b0c: ending tablet copy session 0e2d29c89c754e8ea12ed6730aa4c85d-6da351b3ad134fc78c052d34aafdcbc0 on tablet 6da351b3ad134fc78c052d34aafdcbc0 with peer 0e2d29c89c754e8ea12ed6730aa4c85d Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 34dd0711864545e1beb62517f20f557e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING [LEADER] All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 0e2d29c89c754e8ea12ed6730aa4c85d C = 25bfcaafb29b4431b9ac0f78f5dec9cc The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | [config not available] | | | C | A B C* | 2 | -1 | Yes Tablet 6da351b3ad134fc78c052d34aafdcbc0 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING [LEADER] 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): RUNNING All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | 5 | Yes B | A* B C | 1 | 5 | Yes C | A B C | 1 | 5 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 1 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:31.341907 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:31.602900 521 raft_consensus.cc:1184] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0720 00:42:31.603492 755 consensus_queue.cc:984] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0720 00:42:31.607697 580 consensus_peers.cc:458] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:31.608134 1031 raft_consensus.cc:922] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Attempting to remove follower 0e2d29c89c754e8ea12ed6730aa4c85d from the Raft config. Reason: The tablet replica hosted on peer 0e2d29c89c754e8ea12ed6730aa4c85d has failed I0720 00:42:31.608935 1031 consensus_queue.cc:206] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:31.610262 521 raft_consensus.cc:1184] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0720 00:42:31.610647 755 consensus_queue.cc:984] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:31.612253 1031 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:31.612704 521 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:31.615393 268 catalog_manager.cc:4067] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: config changed from index -1 to 2, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:31.620453 268 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 34dd0711864545e1beb62517f20f557e on 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 2) I0720 00:42:31.620647 268 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet 34dd0711864545e1beb62517f20f557e (attempt 1) I0720 00:42:31.621161 844 tablet_service.cc:1066] Processing DeleteTablet for tablet 34dd0711864545e1beb62517f20f557e with delete_type TABLET_DATA_TOMBSTONED (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:37654 I0720 00:42:31.621846 645 consensus_queue.cc:206] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:31.623215 1064 ts_tablet_manager.cc:1400] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:31.623366 521 raft_consensus.cc:1184] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0720 00:42:31.623805 1031 consensus_queue.cc:984] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s W0720 00:42:31.624025 580 consensus_peers.cc:458] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:31.625571 755 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:31.626144 521 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:31.628744 1064 ts_tablet_manager.cc:1413] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:31.628844 1064 log.cc:990] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/wal/wals/34dd0711864545e1beb62517f20f557e I0720 00:42:31.629510 251 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet 34dd0711864545e1beb62517f20f557e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:31.629647 253 catalog_manager.cc:3437] TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): tablet 34dd0711864545e1beb62517f20f557e (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) successfully deleted I0720 00:42:31.630875 268 catalog_manager.cc:4067] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: config changed from index 2 to 3, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New cstate: current_term: 2 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:31.698791 863 raft_consensus.cc:1126] T 6da351b3ad134fc78c052d34aafdcbc0 P 0e2d29c89c754e8ea12ed6730aa4c85d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[] I0720 00:42:32.021119 755 raft_consensus.cc:922] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Attempting to remove follower 0e2d29c89c754e8ea12ed6730aa4c85d from the Raft config. Reason: The tablet replica hosted on peer 0e2d29c89c754e8ea12ed6730aa4c85d has failed I0720 00:42:32.021652 755 consensus_queue.cc:206] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } I0720 00:42:32.023187 521 raft_consensus.cc:1184] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0720 00:42:32.023654 1031 consensus_queue.cc:984] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:32.025441 755 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:32.025876 521 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:32.028904 268 catalog_manager.cc:4067] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c reported cstate change: config changed from index 3 to 4, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } } I0720 00:42:32.033030 268 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 34dd0711864545e1beb62517f20f557e on 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255) (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 4) I0720 00:42:32.033273 268 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet 34dd0711864545e1beb62517f20f557e (attempt 1) I0720 00:42:32.033792 844 tablet_service.cc:1066] Processing DeleteTablet for tablet 34dd0711864545e1beb62517f20f557e with delete_type TABLET_DATA_TOMBSTONED (TS 0e2d29c89c754e8ea12ed6730aa4c85d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37654 I0720 00:42:32.034476 645 consensus_queue.cc:206] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:32.034824 253 catalog_manager.cc:3437] TS 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): tablet 34dd0711864545e1beb62517f20f557e (table test-workload [id=e0395db4648044fc9c454c18dec46a8e]) successfully deleted I0720 00:42:32.036056 521 raft_consensus.cc:1184] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Refusing update from remote peer 25bfcaafb29b4431b9ac0f78f5dec9cc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) W0720 00:42:32.036203 580 consensus_peers.cc:458] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:32.036466 1031 consensus_queue.cc:984] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0720 00:42:32.038103 755 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:32.038626 521 raft_consensus.cc:2792] T 34dd0711864545e1beb62517f20f557e P 4deaf10768b24896bcd716f79c982b0c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } I0720 00:42:32.040366 251 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet 34dd0711864545e1beb62517f20f557e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:32.041321 268 catalog_manager.cc:4067] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc reported cstate change: config changed from index 4 to 5, VOTER 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129) added. New cstate: current_term: 2 leader_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 34dd0711864545e1beb62517f20f557e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 4deaf10768b24896bcd716f79c982b0c (127.0.58.130:38913): RUNNING 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777): RUNNING [LEADER] 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): not running State: FAILED Data state: TABLET_DATA_TOMBSTONED Last status: Deleted tablet blocks from disk All reported replicas are: A = 4deaf10768b24896bcd716f79c982b0c B = 25bfcaafb29b4431b9ac0f78f5dec9cc C = 0e2d29c89c754e8ea12ed6730aa4c85d The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 2 | 5 | Yes B | A B* C | 2 | 5 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 5 | 4 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:32.172775 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:32.544231 1091 ts_tablet_manager.cc:682] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Initiating tablet copy from peer 25bfcaafb29b4431b9ac0f78f5dec9cc (127.0.58.131:34777) I0720 00:42:32.544322 1091 tablet_copy_client.cc:204] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:32.544464 1091 tablet_copy_client.cc:241] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.131:34777 I0720 00:42:32.549702 667 tablet_copy_service.cc:135] P 25bfcaafb29b4431b9ac0f78f5dec9cc: Received BeginTabletCopySession request for tablet 34dd0711864545e1beb62517f20f557e from peer 0e2d29c89c754e8ea12ed6730aa4c85d ({username='slave'} at 127.0.58.129:49451) I0720 00:42:32.549880 667 tablet_copy_service.cc:156] P 25bfcaafb29b4431b9ac0f78f5dec9cc: Beginning new tablet copy session on tablet 34dd0711864545e1beb62517f20f557e from peer 0e2d29c89c754e8ea12ed6730aa4c85d at {username='slave'} at 127.0.58.129:49451: session id = 0e2d29c89c754e8ea12ed6730aa4c85d-34dd0711864545e1beb62517f20f557e I0720 00:42:32.551944 667 tablet_copy_source_session.cc:202] T 34dd0711864545e1beb62517f20f557e P 25bfcaafb29b4431b9ac0f78f5dec9cc: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:32.553014 1091 ts_tablet_manager.cc:1400] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:42:32.557366 1091 ts_tablet_manager.cc:1413] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:32.559443 1091 tablet_copy_client.cc:579] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Starting download of 0 data blocks... I0720 00:42:32.559644 1091 tablet_copy_client.cc:542] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Starting download of 1 WAL segments... I0720 00:42:32.561120 1091 tablet_copy_client.cc:418] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:32.563097 1091 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap starting. I0720 00:42:32.569038 1091 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:32.569392 1091 tablet_bootstrap.cc:439] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Bootstrap complete. I0720 00:42:32.569548 1091 ts_tablet_manager.cc:1059] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s I0720 00:42:32.570171 1091 raft_consensus.cc:340] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:32.570323 1091 raft_consensus.cc:705] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e2d29c89c754e8ea12ed6730aa4c85d, State: Initialized, Role: FOLLOWER I0720 00:42:32.570511 1091 consensus_queue.cc:229] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4deaf10768b24896bcd716f79c982b0c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 38913 } } peers { permanent_uuid: "25bfcaafb29b4431b9ac0f78f5dec9cc" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 34777 } } peers { permanent_uuid: "0e2d29c89c754e8ea12ed6730aa4c85d" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34255 } } I0720 00:42:32.571063 1091 ts_tablet_manager.cc:1087] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s I0720 00:42:32.571712 667 tablet_copy_service.cc:325] P 25bfcaafb29b4431b9ac0f78f5dec9cc: Request end of tablet copy session 0e2d29c89c754e8ea12ed6730aa4c85d-34dd0711864545e1beb62517f20f557e received from {username='slave'} at 127.0.58.129:49451 I0720 00:42:32.571815 667 tablet_copy_service.cc:413] P 25bfcaafb29b4431b9ac0f78f5dec9cc: ending tablet copy session 0e2d29c89c754e8ea12ed6730aa4c85d-34dd0711864545e1beb62517f20f557e on tablet 34dd0711864545e1beb62517f20f557e with peer 0e2d29c89c754e8ea12ed6730aa4c85d I0720 00:42:33.162243 863 raft_consensus.cc:1126] T 34dd0711864545e1beb62517f20f557e P 0e2d29c89c754e8ea12ed6730aa4c85d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[] Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- d589d9ed25814d82889b1dbe3cdb2b70 | 127.0.58.190:37585 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0e2d29c89c754e8ea12ed6730aa4c85d | 127.0.58.129:34255 | HEALTHY | 25bfcaafb29b4431b9ac0f78f5dec9cc | 127.0.58.131:34777 | HEALTHY | 4deaf10768b24896bcd716f79c982b0c | 127.0.58.130:38913 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:34255 env_inject_eio | 1 | hidden | 127.0.58.129:34255 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34255 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34255 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:38913 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:34777 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34255 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:38913 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:34777 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:42:33.245296 234 log_verifier.cc:121] Checking tablet 34dd0711864545e1beb62517f20f557e I0720 00:42:33.274492 234 log_verifier.cc:172] Verified matching terms for 5 ops in tablet 34dd0711864545e1beb62517f20f557e I0720 00:42:33.274602 234 log_verifier.cc:121] Checking tablet 6da351b3ad134fc78c052d34aafdcbc0 I0720 00:42:33.301017 234 log_verifier.cc:172] Verified matching terms for 5 ops in tablet 6da351b3ad134fc78c052d34aafdcbc0 I0720 00:42:33.301079 234 log_verifier.cc:121] Checking tablet 7cab46396b7e471f89b67a9bee891c6a I0720 00:42:33.325492 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 7cab46396b7e471f89b67a9bee891c6a I0720 00:42:33.325544 234 log_verifier.cc:121] Checking tablet 8668997e728c430da205e7d36533f74a I0720 00:42:33.352324 234 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 8668997e728c430da205e7d36533f74a I0720 00:42:33.352389 234 log_verifier.cc:121] Checking tablet ea63d074a81e47c7910a7097efab544e I0720 00:42:33.378629 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet ea63d074a81e47c7910a7097efab544e I0720 00:42:33.642149 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 775 I0720 00:42:33.656355 906 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:33.806378 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 775 W0720 00:42:33.827761 454 connection.cc:655] client connection to 127.0.58.129:34255 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0720 00:42:33.827985 454 consensus_peers.cc:458] T 6da351b3ad134fc78c052d34aafdcbc0 P 4deaf10768b24896bcd716f79c982b0c -> Peer 0e2d29c89c754e8ea12ed6730aa4c85d (127.0.58.129:34255): Couldn't send request to peer 0e2d29c89c754e8ea12ed6730aa4c85d. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:33.828752 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 441 I0720 00:42:33.835938 561 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:33.927096 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 441 I0720 00:42:33.940480 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 567 I0720 00:42:33.945132 687 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:34.026866 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 567 I0720 00:42:34.040127 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 240 I0720 00:42:34.041030 300 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:34.121618 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 240 [ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/0 (6643 ms) [ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/1 I0720 00:42:34.156183 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-4 --block_manager=file --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:33793 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:34.344051 1138 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:34.344259 1138 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:34.346796 1138 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:42:34.346845 1138 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:42:34.346885 1138 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:34.346912 1138 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:42:34.346946 1138 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:42:34.351459 1138 master_main.cc:98] Master server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:33793 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:34.358026 1138 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 38516us I0720 00:42:34.359364 1138 fs_manager.cc:263] Metadata directory not provided I0720 00:42:34.359419 1138 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:34.359731 1138 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:34.359802 1138 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:34.362236 1138 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.362749 1138 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.363060 1138 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.363344 1138 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-3/instance: uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.363637 1138 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-4/instance: uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.363929 1138 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.374106 1138 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.001s I0720 00:42:34.375522 1138 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:34.376072 1138 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:34.376616 1138 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.002s sys 0.000s I0720 00:42:34.376679 1138 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/wal uuid: "cfcd88079c52477ebcae1d98c42dcb23" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.376823 1138 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:34.419593 1138 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:34.433137 1138 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:33793 I0720 00:42:34.434242 1138 webserver.cc:374] Webserver started at http://127.0.58.190:40339/ using document root and password file I0720 00:42:34.434991 1138 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:42:34.439508 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 1138 I0720 00:42:34.440333 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:42:34.452358 1201 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23: Bootstrap starting. I0720 00:42:34.455128 1201 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23: No blocks or log segments found. Creating new log. I0720 00:42:34.456069 1201 log.cc:525] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23: Log is configured to *not* fsync() on all Append() calls I0720 00:42:34.457772 1201 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23: No bootstrap required, opened a new log I0720 00:42:34.464208 1201 raft_consensus.cc:340] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } I0720 00:42:34.464457 1201 raft_consensus.cc:366] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:34.464540 1201 raft_consensus.cc:705] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcd88079c52477ebcae1d98c42dcb23, State: Initialized, Role: FOLLOWER I0720 00:42:34.465059 1201 consensus_queue.cc:229] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } I0720 00:42:34.465251 1201 raft_consensus.cc:380] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:42:34.465320 1201 raft_consensus.cc:472] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:42:34.465436 1201 raft_consensus.cc:2890] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:34.467133 1201 raft_consensus.cc:494] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } I0720 00:42:34.467574 1201 leader_election.cc:310] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfcd88079c52477ebcae1d98c42dcb23; no voters: I0720 00:42:34.468232 1201 leader_election.cc:296] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:42:34.468652 1204 raft_consensus.cc:2644] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:34.469241 1204 raft_consensus.cc:667] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [term 1 LEADER]: Becoming Leader. State: Replica: cfcd88079c52477ebcae1d98c42dcb23, State: Running, Role: LEADER I0720 00:42:34.469677 1204 consensus_queue.cc:206] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } I0720 00:42:34.472050 1201 sys_catalog.cc:340] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "cfcd88079c52477ebcae1d98c42dcb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } } I0720 00:42:34.471833 1205 sys_catalog.cc:340] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfcd88079c52477ebcae1d98c42dcb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } } I0720 00:42:34.472091 1206 sys_catalog.cc:340] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfcd88079c52477ebcae1d98c42dcb23. Latest consensus state: current_term: 1 leader_uuid: "cfcd88079c52477ebcae1d98c42dcb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfcd88079c52477ebcae1d98c42dcb23" member_type: VOTER } } I0720 00:42:34.472684 1205 sys_catalog.cc:343] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: This master's current role is: LEADER I0720 00:42:34.472740 1206 sys_catalog.cc:343] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: This master's current role is: LEADER I0720 00:42:34.472581 1201 sys_catalog.cc:343] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: This master's current role is: LEADER I0720 00:42:34.474963 1201 sys_catalog.cc:428] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23 [sys.catalog]: configured and running, proceeding with master startup. I0720 00:42:34.475384 1208 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:42:34.477440 1210 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0720 00:42:34.484637 1208 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:42:34.556602 1208 catalog_manager.cc:959] Generated new certificate authority record I0720 00:42:34.558903 1208 catalog_manager.cc:1084] Loading token signing keys... I0720 00:42:34.566654 1208 catalog_manager.cc:4371] T 00000000000000000000000000000000 P cfcd88079c52477ebcae1d98c42dcb23: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:34.616071 1202 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:34.616314 1202 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:34.618415 1202 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:42:34.618479 1202 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:34.618602 1202 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:42:34.634517 1202 tablet_server_main.cc:78] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:34.642380 1202 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 38516us I0720 00:42:34.643131 1202 fs_manager.cc:263] Metadata directory not provided I0720 00:42:34.643196 1202 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:34.643518 1202 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:34.643581 1202 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:34.645844 1202 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.646392 1202 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.646736 1202 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.647054 1202 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3/instance: uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.647393 1202 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4/instance: uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.647711 1202 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.673144 1202 fs_manager.cc:503] Time spent creating directory manager: real 0.025s user 0.011s sys 0.012s I0720 00:42:34.675282 1202 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:34.675976 1202 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:34.676620 1202 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.002s sys 0.000s I0720 00:42:34.676704 1202 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.676903 1202 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:34.709553 1202 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:34.711661 1202 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:34.711746 1202 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:34.738495 1202 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:33685 I0720 00:42:34.739753 1202 webserver.cc:374] Webserver started at http://127.0.58.129:38721/ using document root and password file I0720 00:42:34.740445 1202 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:42:34.746192 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 1202 I0720 00:42:34.747021 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:42:34.758846 1335 heartbeater.cc:324] Connected to a master server at 127.0.58.190:33793 I0720 00:42:34.759110 1335 heartbeater.cc:405] Registering TS with master... I0720 00:42:34.759742 1335 heartbeater.cc:454] Master 127.0.58.190:33793 requested a full tablet report, sending... I0720 00:42:34.762007 1168 ts_manager.cc:161] Registered new tserver with Master: 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) I0720 00:42:34.764338 1168 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:58267 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:34.905220 1339 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:34.905464 1339 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:34.907562 1339 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:42:34.907627 1339 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:34.907732 1339 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:42:34.911541 1339 tablet_server_main.cc:78] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:34.919287 1339 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 38516us I0720 00:42:34.920028 1339 fs_manager.cc:263] Metadata directory not provided I0720 00:42:34.920094 1339 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:34.920411 1339 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:34.920475 1339 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:34.922854 1339 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.923393 1339 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.923738 1339 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.924051 1339 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-3/instance: uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.924387 1339 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-4/instance: uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.924705 1339 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.934780 1339 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.010s sys 0.001s I0720 00:42:34.936153 1339 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:34.936679 1339 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:34.937259 1339 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.003s sys 0.000s I0720 00:42:34.937330 1339 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/wal uuid: "f554abf9579c42029052fc648fb4bd8c" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:34.937475 1339 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:35.003644 1339 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:35.005200 1339 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:35.005285 1339 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:35.026942 1339 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:45963 I0720 00:42:35.028072 1339 webserver.cc:374] Webserver started at http://127.0.58.130:33579/ using document root and password file I0720 00:42:35.028715 1339 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:42:35.035126 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 1339 I0720 00:42:35.035984 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:42:35.052745 1462 heartbeater.cc:324] Connected to a master server at 127.0.58.190:33793 I0720 00:42:35.052966 1462 heartbeater.cc:405] Registering TS with master... I0720 00:42:35.053453 1462 heartbeater.cc:454] Master 127.0.58.190:33793 requested a full tablet report, sending... I0720 00:42:35.054549 1168 ts_manager.cc:161] Registered new tserver with Master: f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963) I0720 00:42:35.055361 1168 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:57291 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:35.193539 1465 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:35.193750 1465 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:35.195911 1465 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:42:35.195963 1465 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:35.196056 1465 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:42:35.200512 1465 tablet_server_main.cc:78] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:35.208113 1465 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39016us I0720 00:42:35.208811 1465 fs_manager.cc:263] Metadata directory not provided I0720 00:42:35.208866 1465 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:35.209172 1465 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:35.209229 1465 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:35.211457 1465 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.211941 1465 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.212275 1465 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.212550 1465 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-3/instance: uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.212859 1465 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-4/instance: uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.213138 1465 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.223258 1465 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.011s sys 0.001s I0720 00:42:35.224664 1465 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:35.225179 1465 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:35.225811 1465 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.002s sys 0.001s I0720 00:42:35.225878 1465 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/wal uuid: "da17ba98538b41b091e9cbf779bc5afa" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.226044 1465 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:35.247064 1465 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:35.248577 1465 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:35.248644 1465 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:35.273629 1465 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:39787 I0720 00:42:35.274783 1465 webserver.cc:374] Webserver started at http://127.0.58.131:42567/ using document root and password file I0720 00:42:35.275436 1465 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:42:35.281543 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 1465 I0720 00:42:35.282219 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.132:0 --local_ip_for_outbound_sockets=127.0.58.132 --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:42:35.288905 1588 heartbeater.cc:324] Connected to a master server at 127.0.58.190:33793 I0720 00:42:35.289119 1588 heartbeater.cc:405] Registering TS with master... I0720 00:42:35.289654 1588 heartbeater.cc:454] Master 127.0.58.190:33793 requested a full tablet report, sending... I0720 00:42:35.290884 1168 ts_manager.cc:161] Registered new tserver with Master: da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787) I0720 00:42:35.292006 1168 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:51893 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:35.435431 1591 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:35.435637 1591 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:35.437671 1591 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:42:35.437721 1591 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:35.437817 1591 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.132 I0720 00:42:35.441398 1591 tablet_server_main.cc:78] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:33793 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.132 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:35.449376 1591 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39016us I0720 00:42:35.450151 1591 fs_manager.cc:263] Metadata directory not provided I0720 00:42:35.450211 1591 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:35.450512 1591 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:35.450569 1591 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:35.452782 1591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/instance: uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.453277 1591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-1/instance: uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.453583 1591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-2/instance: uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.453855 1591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-3/instance: uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.454176 1591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-4/instance: uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.454470 1591 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/wal/instance: uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.464671 1591 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.001s I0720 00:42:35.466043 1591 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:35.466584 1591 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:35.467141 1591 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0720 00:42:35.467221 1591 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/wal uuid: "0eeb775c53b34d00ab90623d4b54577f" format_stamp: "Formatted at 2019-07-20 00:42:35 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:35.467360 1591 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:35.515492 1591 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:35.517029 1591 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:35.517094 1591 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:35.541136 1591 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.132:39821 I0720 00:42:35.542281 1591 webserver.cc:374] Webserver started at http://127.0.58.132:45991/ using document root and password file I0720 00:42:35.542899 1591 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb I0720 00:42:35.546219 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 1591 I0720 00:42:35.558990 1714 heartbeater.cc:324] Connected to a master server at 127.0.58.190:33793 I0720 00:42:35.559197 1714 heartbeater.cc:405] Registering TS with master... I0720 00:42:35.559649 1714 heartbeater.cc:454] Master 127.0.58.190:33793 requested a full tablet report, sending... I0720 00:42:35.560814 1168 ts_manager.cc:161] Registered new tserver with Master: 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821) I0720 00:42:35.561579 1168 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.132:34705 I0720 00:42:35.562737 234 external_mini_cluster.cc:628] 4 TS(s) registered with all masters I0720 00:42:35.576391 234 test_util.cc:212] Using random seed: 511467564 I0720 00:42:35.591223 1168 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40312: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff" } partition_schema { range_schema { columns { name: "key" } } } I0720 00:42:35.625057 1399 tablet_service.cc:1022] Processing CreateTablet for tablet 1aa1fb28c3d546e88c2d549fe18c1b76 (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:35.625638 1651 tablet_service.cc:1022] Processing CreateTablet for tablet 1aa1fb28c3d546e88c2d549fe18c1b76 (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:35.627573 1271 tablet_service.cc:1022] Processing CreateTablet for tablet c362ba31e9ec40999c7c048ce93e99ff (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:35.629261 1650 tablet_service.cc:1022] Processing CreateTablet for tablet c362ba31e9ec40999c7c048ce93e99ff (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:35.630455 1270 tablet_service.cc:1022] Processing CreateTablet for tablet 8b450357aeba4ea4b2b282750617b863 (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:35.630762 1398 tablet_service.cc:1022] Processing CreateTablet for tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:35.630916 1649 tablet_service.cc:1022] Processing CreateTablet for tablet 8b450357aeba4ea4b2b282750617b863 (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:35.630766 1524 tablet_service.cc:1022] Processing CreateTablet for tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:35.631526 1269 tablet_service.cc:1022] Processing CreateTablet for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:35.632781 1272 tablet_service.cc:1022] Processing CreateTablet for tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:35.633641 1397 tablet_service.cc:1022] Processing CreateTablet for tablet c362ba31e9ec40999c7c048ce93e99ff (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:35.634460 1523 tablet_service.cc:1022] Processing CreateTablet for tablet 8b450357aeba4ea4b2b282750617b863 (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:35.635541 1522 tablet_service.cc:1022] Processing CreateTablet for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:35.637228 1525 tablet_service.cc:1022] Processing CreateTablet for tablet 1aa1fb28c3d546e88c2d549fe18c1b76 (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:35.641317 1396 tablet_service.cc:1022] Processing CreateTablet for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (table=test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:35.663182 1742 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: Bootstrap starting. I0720 00:42:35.679833 1749 tablet_bootstrap.cc:439] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa: Bootstrap starting. I0720 00:42:35.679922 1741 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f: Bootstrap starting. I0720 00:42:35.682525 1743 tablet_bootstrap.cc:439] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: Bootstrap starting. I0720 00:42:35.682742 1748 tablet_bootstrap.cc:439] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c: Bootstrap starting. I0720 00:42:35.684008 1743 tablet_bootstrap.cc:590] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: No blocks or log segments found. Creating new log. I0720 00:42:35.685093 1743 log.cc:525] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: Log is configured to *not* fsync() on all Append() calls I0720 00:42:35.685500 1748 tablet_bootstrap.cc:590] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c: No blocks or log segments found. Creating new log. I0720 00:42:35.686419 1742 tablet_bootstrap.cc:590] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: No blocks or log segments found. Creating new log. I0720 00:42:35.686560 1745 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: Bootstrap starting. I0720 00:42:35.688683 1745 tablet_bootstrap.cc:590] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: No blocks or log segments found. Creating new log. I0720 00:42:35.689793 1745 log.cc:525] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: Log is configured to *not* fsync() on all Append() calls I0720 00:42:35.691810 1741 tablet_bootstrap.cc:590] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f: No blocks or log segments found. Creating new log. I0720 00:42:35.691985 1744 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa: Bootstrap starting. I0720 00:42:35.693591 1744 tablet_bootstrap.cc:590] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa: No blocks or log segments found. Creating new log. I0720 00:42:35.694509 1756 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c: Bootstrap starting. I0720 00:42:35.696163 1756 tablet_bootstrap.cc:590] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c: No blocks or log segments found. Creating new log. I0720 00:42:35.696492 1742 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: No bootstrap required, opened a new log I0720 00:42:35.697088 1756 log.cc:525] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c: Log is configured to *not* fsync() on all Append() calls I0720 00:42:35.697381 1742 ts_tablet_manager.cc:1059] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.004s I0720 00:42:35.698696 1741 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f: No bootstrap required, opened a new log I0720 00:42:35.698817 1741 ts_tablet_manager.cc:1059] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s I0720 00:42:35.700038 1750 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:35.703284 1752 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa: Bootstrap starting. I0720 00:42:35.703907 1756 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c: No bootstrap required, opened a new log I0720 00:42:35.704197 1756 ts_tablet_manager.cc:1059] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s I0720 00:42:35.704573 1752 tablet_bootstrap.cc:590] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa: No blocks or log segments found. Creating new log. I0720 00:42:35.705289 1758 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:35.707041 1758 tablet_bootstrap.cc:590] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: No blocks or log segments found. Creating new log. I0720 00:42:35.708356 1758 log.cc:525] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Log is configured to *not* fsync() on all Append() calls I0720 00:42:35.715145 1751 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c: Bootstrap starting. I0720 00:42:35.715790 1747 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:35.717017 1747 tablet_bootstrap.cc:590] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: No blocks or log segments found. Creating new log. I0720 00:42:35.718708 1757 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:35.719442 1751 tablet_bootstrap.cc:590] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c: No blocks or log segments found. Creating new log. I0720 00:42:35.719380 1741 raft_consensus.cc:340] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.719857 1741 raft_consensus.cc:366] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.719930 1741 raft_consensus.cc:705] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeb775c53b34d00ab90623d4b54577f, State: Initialized, Role: FOLLOWER I0720 00:42:35.720096 1757 tablet_bootstrap.cc:590] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: No blocks or log segments found. Creating new log. I0720 00:42:35.720449 1741 consensus_queue.cc:229] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.721050 1742 raft_consensus.cc:340] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.721253 1742 raft_consensus.cc:366] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.721310 1742 raft_consensus.cc:705] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeb775c53b34d00ab90623d4b54577f, State: Initialized, Role: FOLLOWER I0720 00:42:35.721504 1742 consensus_queue.cc:229] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.722383 1714 heartbeater.cc:446] Master 127.0.58.190:33793 was elected leader, sending a full tablet report... I0720 00:42:35.722677 1749 tablet_bootstrap.cc:590] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa: No blocks or log segments found. Creating new log. I0720 00:42:35.723832 1746 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: Bootstrap starting. I0720 00:42:35.723673 1756 raft_consensus.cc:340] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.725503 1752 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa: No bootstrap required, opened a new log I0720 00:42:35.725832 1752 ts_tablet_manager.cc:1059] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s I0720 00:42:35.726543 1750 tablet_bootstrap.cc:590] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: No blocks or log segments found. Creating new log. I0720 00:42:35.726742 1756 raft_consensus.cc:366] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.726851 1756 raft_consensus.cc:705] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f554abf9579c42029052fc648fb4bd8c, State: Initialized, Role: FOLLOWER I0720 00:42:35.725697 1746 tablet_bootstrap.cc:590] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: No blocks or log segments found. Creating new log. I0720 00:42:35.726641 1751 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c: No bootstrap required, opened a new log I0720 00:42:35.727820 1751 ts_tablet_manager.cc:1059] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0720 00:42:35.728677 1744 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa: No bootstrap required, opened a new log I0720 00:42:35.728804 1744 ts_tablet_manager.cc:1059] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s I0720 00:42:35.729313 1751 raft_consensus.cc:340] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.729591 1751 raft_consensus.cc:366] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.729713 1751 raft_consensus.cc:705] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f554abf9579c42029052fc648fb4bd8c, State: Initialized, Role: FOLLOWER I0720 00:42:35.729943 1751 consensus_queue.cc:229] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.730427 1756 consensus_queue.cc:229] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.731050 1743 tablet_bootstrap.cc:439] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: No bootstrap required, opened a new log I0720 00:42:35.731220 1743 ts_tablet_manager.cc:1059] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.012s I0720 00:42:35.731981 1743 raft_consensus.cc:340] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.732261 1743 raft_consensus.cc:366] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.732313 1743 raft_consensus.cc:705] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eeb775c53b34d00ab90623d4b54577f, State: Initialized, Role: FOLLOWER I0720 00:42:35.732611 1743 consensus_queue.cc:229] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.733424 1743 ts_tablet_manager.cc:1087] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0720 00:42:35.732920 1742 ts_tablet_manager.cc:1087] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: Time spent starting tablet: real 0.034s user 0.003s sys 0.003s I0720 00:42:35.734566 1462 heartbeater.cc:446] Master 127.0.58.190:33793 was elected leader, sending a full tablet report... I0720 00:42:35.735088 1749 tablet_bootstrap.cc:439] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa: No bootstrap required, opened a new log I0720 00:42:35.735224 1749 ts_tablet_manager.cc:1059] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s I0720 00:42:35.736591 1741 ts_tablet_manager.cc:1087] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f: Time spent starting tablet: real 0.038s user 0.012s sys 0.000s I0720 00:42:35.737313 1757 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: No bootstrap required, opened a new log I0720 00:42:35.737709 1757 ts_tablet_manager.cc:1059] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s I0720 00:42:35.738790 1758 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: No bootstrap required, opened a new log I0720 00:42:35.738946 1758 ts_tablet_manager.cc:1059] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s I0720 00:42:35.739759 1745 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: No bootstrap required, opened a new log I0720 00:42:35.739905 1745 ts_tablet_manager.cc:1059] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s I0720 00:42:35.740423 1746 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: No bootstrap required, opened a new log I0720 00:42:35.740550 1746 ts_tablet_manager.cc:1059] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s I0720 00:42:35.740829 1751 ts_tablet_manager.cc:1087] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s I0720 00:42:35.741156 1750 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: No bootstrap required, opened a new log I0720 00:42:35.741281 1750 ts_tablet_manager.cc:1059] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s I0720 00:42:35.742156 1756 ts_tablet_manager.cc:1087] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c: Time spent starting tablet: real 0.038s user 0.006s sys 0.010s I0720 00:42:35.741837 1746 raft_consensus.cc:340] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.742358 1746 raft_consensus.cc:366] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.742404 1748 tablet_bootstrap.cc:439] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c: No bootstrap required, opened a new log I0720 00:42:35.742580 1748 ts_tablet_manager.cc:1059] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.002s I0720 00:42:35.742760 1747 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: No bootstrap required, opened a new log I0720 00:42:35.742866 1747 ts_tablet_manager.cc:1059] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s I0720 00:42:35.743305 1748 raft_consensus.cc:340] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.743528 1748 raft_consensus.cc:366] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.743592 1748 raft_consensus.cc:705] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f554abf9579c42029052fc648fb4bd8c, State: Initialized, Role: FOLLOWER I0720 00:42:35.743753 1748 consensus_queue.cc:229] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.744554 1748 ts_tablet_manager.cc:1087] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0720 00:42:35.742429 1746 raft_consensus.cc:705] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f554abf9579c42029052fc648fb4bd8c, State: Initialized, Role: FOLLOWER I0720 00:42:35.744827 1746 consensus_queue.cc:229] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.745214 1746 ts_tablet_manager.cc:1087] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s I0720 00:42:35.746482 1757 raft_consensus.cc:340] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.746968 1757 raft_consensus.cc:366] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.747057 1757 raft_consensus.cc:705] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: FOLLOWER I0720 00:42:35.747229 1750 raft_consensus.cc:340] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.747454 1750 raft_consensus.cc:366] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.747514 1750 raft_consensus.cc:705] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: FOLLOWER I0720 00:42:35.747555 1757 consensus_queue.cc:229] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.747673 1750 consensus_queue.cc:229] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.748062 1747 raft_consensus.cc:340] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.748275 1747 raft_consensus.cc:366] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.748139 1758 raft_consensus.cc:340] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.748387 1758 raft_consensus.cc:366] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.748435 1758 raft_consensus.cc:705] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: FOLLOWER I0720 00:42:35.749087 1335 heartbeater.cc:446] Master 127.0.58.190:33793 was elected leader, sending a full tablet report... I0720 00:42:35.749764 1747 raft_consensus.cc:705] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: FOLLOWER I0720 00:42:35.750011 1757 ts_tablet_manager.cc:1087] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s I0720 00:42:35.750217 1750 ts_tablet_manager.cc:1087] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s I0720 00:42:35.750188 1758 consensus_queue.cc:229] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.750355 1747 consensus_queue.cc:229] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.750862 1747 ts_tablet_manager.cc:1087] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s I0720 00:42:35.751699 1758 ts_tablet_manager.cc:1087] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s I0720 00:42:35.752243 1772 raft_consensus.cc:472] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:35.752409 1772 raft_consensus.cc:494] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.754498 1772 leader_election.cc:296] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787), 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821) I0720 00:42:35.761893 1545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1aa1fb28c3d546e88c2d549fe18c1b76" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da17ba98538b41b091e9cbf779bc5afa" is_pre_election: true W0720 00:42:35.762933 1352 leader_election.cc:348] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): Illegal state: must be running to vote when last-logged opid is not known I0720 00:42:35.764081 1671 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1aa1fb28c3d546e88c2d549fe18c1b76" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eeb775c53b34d00ab90623d4b54577f" is_pre_election: true I0720 00:42:35.764466 1671 raft_consensus.cc:2350] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f554abf9579c42029052fc648fb4bd8c in term 0. I0720 00:42:35.764914 1353 leader_election.cc:310] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0eeb775c53b34d00ab90623d4b54577f, f554abf9579c42029052fc648fb4bd8c; no voters: da17ba98538b41b091e9cbf779bc5afa I0720 00:42:35.764768 1744 raft_consensus.cc:340] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.765187 1744 raft_consensus.cc:366] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.765264 1744 raft_consensus.cc:705] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da17ba98538b41b091e9cbf779bc5afa, State: Initialized, Role: FOLLOWER I0720 00:42:35.765357 1772 raft_consensus.cc:2644] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:35.765453 1772 raft_consensus.cc:472] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:35.765511 1772 raft_consensus.cc:2890] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.765712 1744 consensus_queue.cc:229] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.766247 1749 raft_consensus.cc:340] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.766425 1749 raft_consensus.cc:366] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.766471 1749 raft_consensus.cc:705] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da17ba98538b41b091e9cbf779bc5afa, State: Initialized, Role: FOLLOWER I0720 00:42:35.766688 1745 raft_consensus.cc:340] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.766867 1745 raft_consensus.cc:366] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.766904 1745 raft_consensus.cc:705] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da17ba98538b41b091e9cbf779bc5afa, State: Initialized, Role: FOLLOWER I0720 00:42:35.767058 1745 consensus_queue.cc:229] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.767441 1752 raft_consensus.cc:340] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.767488 1772 raft_consensus.cc:494] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.767609 1752 raft_consensus.cc:366] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:35.767652 1752 raft_consensus.cc:705] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da17ba98538b41b091e9cbf779bc5afa, State: Initialized, Role: FOLLOWER I0720 00:42:35.767833 1752 consensus_queue.cc:229] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.768173 1749 consensus_queue.cc:229] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.768462 1671 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1aa1fb28c3d546e88c2d549fe18c1b76" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eeb775c53b34d00ab90623d4b54577f" I0720 00:42:35.768512 1545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "1aa1fb28c3d546e88c2d549fe18c1b76" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da17ba98538b41b091e9cbf779bc5afa" I0720 00:42:35.768646 1671 raft_consensus.cc:2890] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.768682 1545 raft_consensus.cc:2890] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.771062 1772 leader_election.cc:296] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 election: Requested vote from peers da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787), 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821) I0720 00:42:35.771756 1588 heartbeater.cc:446] Master 127.0.58.190:33793 was elected leader, sending a full tablet report... I0720 00:42:35.772505 1749 ts_tablet_manager.cc:1087] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s I0720 00:42:35.772943 1752 ts_tablet_manager.cc:1087] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa: Time spent starting tablet: real 0.047s user 0.000s sys 0.008s I0720 00:42:35.773031 1745 ts_tablet_manager.cc:1087] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s I0720 00:42:35.773063 1744 ts_tablet_manager.cc:1087] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa: Time spent starting tablet: real 0.044s user 0.005s sys 0.005s W0720 00:42:35.777529 1589 tablet.cc:1799] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:42:35.781002 1463 tablet.cc:1799] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:42:35.794951 1715 tablet.cc:1799] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:35.806211 1782 raft_consensus.cc:472] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:35.806392 1782 raft_consensus.cc:494] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.807740 1782 leader_election.cc:296] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821), 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) I0720 00:42:35.822029 1670 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b450357aeba4ea4b2b282750617b863" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eeb775c53b34d00ab90623d4b54577f" is_pre_election: true I0720 00:42:35.822057 1292 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b450357aeba4ea4b2b282750617b863" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" is_pre_election: true I0720 00:42:35.822331 1670 raft_consensus.cc:2350] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 0. I0720 00:42:35.822454 1292 raft_consensus.cc:2350] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 0. I0720 00:42:35.822772 1479 leader_election.cc:310] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eeb775c53b34d00ab90623d4b54577f, da17ba98538b41b091e9cbf779bc5afa; no voters: I0720 00:42:35.823220 1782 raft_consensus.cc:2644] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:35.823298 1782 raft_consensus.cc:472] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:35.823356 1782 raft_consensus.cc:2890] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.828773 1771 raft_consensus.cc:472] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:35.828958 1771 raft_consensus.cc:494] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.830265 1771 leader_election.cc:296] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21544559ee284906bc8977db1465db7b (127.0.58.129:33685), f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963) I0720 00:42:35.834532 1671 raft_consensus.cc:2350] T 1aa1fb28c3d546e88c2d549fe18c1b76 P 0eeb775c53b34d00ab90623d4b54577f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f554abf9579c42029052fc648fb4bd8c in term 1. I0720 00:42:35.834537 1545 raft_consensus.cc:2350] T 1aa1fb28c3d546e88c2d549fe18c1b76 P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f554abf9579c42029052fc648fb4bd8c in term 1. I0720 00:42:35.835011 1353 leader_election.cc:310] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eeb775c53b34d00ab90623d4b54577f, f554abf9579c42029052fc648fb4bd8c; no voters: I0720 00:42:35.835397 1772 raft_consensus.cc:2644] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:35.835520 1772 raft_consensus.cc:667] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Becoming Leader. State: Replica: f554abf9579c42029052fc648fb4bd8c, State: Running, Role: LEADER I0720 00:42:35.835978 1772 consensus_queue.cc:206] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.838726 1419 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c362ba31e9ec40999c7c048ce93e99ff" candidate_uuid: "0eeb775c53b34d00ab90623d4b54577f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f554abf9579c42029052fc648fb4bd8c" is_pre_election: true I0720 00:42:35.839210 1419 raft_consensus.cc:2350] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eeb775c53b34d00ab90623d4b54577f in term 0. I0720 00:42:35.839721 1603 leader_election.cc:310] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eeb775c53b34d00ab90623d4b54577f, f554abf9579c42029052fc648fb4bd8c; no voters: I0720 00:42:35.840154 1771 raft_consensus.cc:2644] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:35.840239 1771 raft_consensus.cc:472] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:35.840294 1771 raft_consensus.cc:2890] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.844415 1168 catalog_manager.cc:4067] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c reported cstate change: term changed from 0 to 1, leader changed from to f554abf9579c42029052fc648fb4bd8c (127.0.58.130). New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } health_report { overall_health: UNKNOWN } } } I0720 00:42:35.846351 1292 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c362ba31e9ec40999c7c048ce93e99ff" candidate_uuid: "0eeb775c53b34d00ab90623d4b54577f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" is_pre_election: true I0720 00:42:35.846524 1292 raft_consensus.cc:2350] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eeb775c53b34d00ab90623d4b54577f in term 0. I0720 00:42:35.870427 1771 raft_consensus.cc:494] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.870427 1782 raft_consensus.cc:494] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.871234 1771 leader_election.cc:296] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [CANDIDATE]: Term 1 election: Requested vote from peers 21544559ee284906bc8977db1465db7b (127.0.58.129:33685), f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963) I0720 00:42:35.872153 1292 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c362ba31e9ec40999c7c048ce93e99ff" candidate_uuid: "0eeb775c53b34d00ab90623d4b54577f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" I0720 00:42:35.872256 1419 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c362ba31e9ec40999c7c048ce93e99ff" candidate_uuid: "0eeb775c53b34d00ab90623d4b54577f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f554abf9579c42029052fc648fb4bd8c" I0720 00:42:35.872356 1292 raft_consensus.cc:2890] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.872433 1419 raft_consensus.cc:2890] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.872452 1671 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b450357aeba4ea4b2b282750617b863" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eeb775c53b34d00ab90623d4b54577f" I0720 00:42:35.872607 1671 raft_consensus.cc:2890] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.874832 1291 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "8b450357aeba4ea4b2b282750617b863" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" I0720 00:42:35.875043 1291 raft_consensus.cc:2890] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.875416 1782 leader_election.cc:296] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 election: Requested vote from peers 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821), 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) I0720 00:42:35.942050 1782 raft_consensus.cc:472] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:35.942260 1782 raft_consensus.cc:494] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.943657 1782 leader_election.cc:296] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21544559ee284906bc8977db1465db7b (127.0.58.129:33685), f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963) I0720 00:42:35.943924 1290 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5aa1a45e61fa4786a8f6e49ac00c36fd" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" is_pre_election: true I0720 00:42:35.944154 1290 raft_consensus.cc:2350] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 0. I0720 00:42:35.944576 1478 leader_election.cc:310] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21544559ee284906bc8977db1465db7b, da17ba98538b41b091e9cbf779bc5afa; no voters: I0720 00:42:35.944869 1782 raft_consensus.cc:2644] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:35.944943 1782 raft_consensus.cc:472] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:35.945006 1782 raft_consensus.cc:2890] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.949292 1418 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5aa1a45e61fa4786a8f6e49ac00c36fd" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f554abf9579c42029052fc648fb4bd8c" is_pre_election: true I0720 00:42:35.949517 1418 raft_consensus.cc:2350] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 0. I0720 00:42:35.957947 1772 raft_consensus.cc:472] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:35.958106 1772 raft_consensus.cc:494] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.958729 1772 leader_election.cc:296] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787), 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) I0720 00:42:35.959043 1419 raft_consensus.cc:2350] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eeb775c53b34d00ab90623d4b54577f in term 1. I0720 00:42:35.959065 1671 raft_consensus.cc:2350] T 8b450357aeba4ea4b2b282750617b863 P 0eeb775c53b34d00ab90623d4b54577f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 1. I0720 00:42:35.959574 1479 leader_election.cc:310] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eeb775c53b34d00ab90623d4b54577f, da17ba98538b41b091e9cbf779bc5afa; no voters: I0720 00:42:35.960044 1545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a6d7c8f8b6340fab210da8b3907aa6e" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da17ba98538b41b091e9cbf779bc5afa" is_pre_election: true I0720 00:42:35.960110 1603 leader_election.cc:310] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eeb775c53b34d00ab90623d4b54577f, f554abf9579c42029052fc648fb4bd8c; no voters: I0720 00:42:35.960249 1545 raft_consensus.cc:2350] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f554abf9579c42029052fc648fb4bd8c in term 0. I0720 00:42:35.960429 1771 raft_consensus.cc:2644] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:35.960613 1771 raft_consensus.cc:667] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 1 LEADER]: Becoming Leader. State: Replica: 0eeb775c53b34d00ab90623d4b54577f, State: Running, Role: LEADER I0720 00:42:35.960654 1352 leader_election.cc:310] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da17ba98538b41b091e9cbf779bc5afa, f554abf9579c42029052fc648fb4bd8c; no voters: I0720 00:42:35.960913 1772 raft_consensus.cc:2644] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:35.960959 1772 raft_consensus.cc:472] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:35.961015 1772 raft_consensus.cc:2890] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.961040 1771 consensus_queue.cc:206] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:35.961928 1291 raft_consensus.cc:2350] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 1. I0720 00:42:35.962482 1292 raft_consensus.cc:2350] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eeb775c53b34d00ab90623d4b54577f in term 1. I0720 00:42:35.963054 1772 raft_consensus.cc:494] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.964689 1772 leader_election.cc:296] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 election: Requested vote from peers da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787), 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) I0720 00:42:35.964956 1545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a6d7c8f8b6340fab210da8b3907aa6e" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da17ba98538b41b091e9cbf779bc5afa" I0720 00:42:35.965143 1545 raft_consensus.cc:2890] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.966473 1780 raft_consensus.cc:2644] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:35.966706 1780 raft_consensus.cc:667] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [term 1 LEADER]: Becoming Leader. State: Replica: da17ba98538b41b091e9cbf779bc5afa, State: Running, Role: LEADER I0720 00:42:35.967226 1780 consensus_queue.cc:206] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:35.967485 1782 raft_consensus.cc:494] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:35.968535 1782 leader_election.cc:296] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 election: Requested vote from peers 21544559ee284906bc8977db1465db7b (127.0.58.129:33685), f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963) I0720 00:42:35.968498 1292 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5aa1a45e61fa4786a8f6e49ac00c36fd" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" I0720 00:42:35.968706 1292 raft_consensus.cc:2890] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.968983 1419 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5aa1a45e61fa4786a8f6e49ac00c36fd" candidate_uuid: "da17ba98538b41b091e9cbf779bc5afa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f554abf9579c42029052fc648fb4bd8c" I0720 00:42:35.969175 1419 raft_consensus.cc:2890] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.970422 1290 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a6d7c8f8b6340fab210da8b3907aa6e" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" I0720 00:42:35.970573 1290 raft_consensus.cc:2890] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:35.970901 1291 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7a6d7c8f8b6340fab210da8b3907aa6e" candidate_uuid: "f554abf9579c42029052fc648fb4bd8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21544559ee284906bc8977db1465db7b" is_pre_election: true I0720 00:42:35.971737 1168 catalog_manager.cc:4067] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f reported cstate change: term changed from 0 to 1, leader changed from to 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132). New cstate: current_term: 1 leader_uuid: "0eeb775c53b34d00ab90623d4b54577f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } health_report { overall_health: HEALTHY } } } I0720 00:42:35.972527 1167 catalog_manager.cc:4067] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa reported cstate change: term changed from 0 to 1, leader changed from to da17ba98538b41b091e9cbf779bc5afa (127.0.58.131). New cstate: current_term: 1 leader_uuid: "da17ba98538b41b091e9cbf779bc5afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } health_report { overall_health: UNKNOWN } } } W0720 00:42:35.995702 1336 tablet.cc:1799] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:36.044317 1419 raft_consensus.cc:2350] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 1. I0720 00:42:36.044358 1545 raft_consensus.cc:2350] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f554abf9579c42029052fc648fb4bd8c in term 1. I0720 00:42:36.044986 1477 leader_election.cc:310] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da17ba98538b41b091e9cbf779bc5afa, f554abf9579c42029052fc648fb4bd8c; no voters: I0720 00:42:36.045054 1292 raft_consensus.cc:2350] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da17ba98538b41b091e9cbf779bc5afa in term 1. I0720 00:42:36.045424 1352 leader_election.cc:310] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: da17ba98538b41b091e9cbf779bc5afa, f554abf9579c42029052fc648fb4bd8c; no voters: I0720 00:42:36.045759 1772 raft_consensus.cc:2644] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:36.045966 1772 raft_consensus.cc:667] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Becoming Leader. State: Replica: f554abf9579c42029052fc648fb4bd8c, State: Running, Role: LEADER I0720 00:42:36.046311 1290 raft_consensus.cc:2350] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f554abf9579c42029052fc648fb4bd8c in term 1. I0720 00:42:36.046226 1772 consensus_queue.cc:206] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:36.046535 1780 raft_consensus.cc:2644] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:36.046708 1780 raft_consensus.cc:667] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 1 LEADER]: Becoming Leader. State: Replica: da17ba98538b41b091e9cbf779bc5afa, State: Running, Role: LEADER I0720 00:42:36.046981 1291 raft_consensus.cc:2325] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f554abf9579c42029052fc648fb4bd8c for term 1 because replica is already servicing an update from a current leader or another vote. I0720 00:42:36.046974 1780 consensus_queue.cc:206] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:36.050091 1167 catalog_manager.cc:4067] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c reported cstate change: term changed from 0 to 1, leader changed from to f554abf9579c42029052fc648fb4bd8c (127.0.58.130). New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } health_report { overall_health: UNKNOWN } } } I0720 00:42:36.050338 1168 catalog_manager.cc:4067] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa reported cstate change: term changed from 0 to 1, leader changed from to da17ba98538b41b091e9cbf779bc5afa (127.0.58.131). New cstate: current_term: 1 leader_uuid: "da17ba98538b41b091e9cbf779bc5afa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } } I0720 00:42:36.106312 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 1202 I0720 00:42:36.106914 1332 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:36.161449 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 1202 I0720 00:42:36.172317 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=file --rpc_bind_addresses=127.0.58.129:33685 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_port=38721 --webserver_interface=127.0.58.129 --tserver_master_addrs=127.0.58.190:33793 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** --env_inject_eio=1.0 --crash_on_eio=false --raft_prepare_replacement_before_eviction=true with env {} I0720 00:42:36.232393 1772 consensus_queue.cc:984] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:36.244467 1770 consensus_queue.cc:984] T 1aa1fb28c3d546e88c2d549fe18c1b76 P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:36.334223 1797 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:36.334478 1797 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:36.336648 1797 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:42:36.336700 1797 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:36.336803 1797 flags.cc:404] Enabled experimental flag: --crash_on_eio=false W0720 00:42:36.336872 1797 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:42:36.340792 1797 tablet_server_main.cc:78] Tablet server non-default flags: --block_manager=file --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:33685 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=38721 --tserver_master_addrs=127.0.58.190:33793 --env_inject_eio=1 --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:36.348889 1797 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 39516us W0720 00:42:36.349213 1797 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:42:36.349256 1797 fault_injection.cc:45] THIS SERVER MAY CRASH! I0720 00:42:36.349911 1797 fs_manager.cc:263] Metadata directory not provided I0720 00:42:36.349997 1797 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory E0720 00:42:36.351382 1797 env_posix.cc:316] I/O error, context: INJECTED FAILURE E0720 00:42:36.353209 1797 env_posix.cc:316] I/O error, context: INJECTED FAILURE I0720 00:42:36.353273 1797 block_manager_util.cc:139] Instance is unhealthy: IO error: Failed to read metadata file from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/data/block_manager_instance: INJECTED FAILURE: Input/output error (error 5) I0720 00:42:36.359027 1797 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.007s sys 0.001s I0720 00:42:36.359880 1797 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:36.360410 1797 file_cache.cc:465] Constructed file cache fbm with capacity 419430 I0720 00:42:36.360978 1797 fs_manager.cc:419] Time spent opening block manager: real 0.000s user 0.003s sys 0.000s I0720 00:42:36.361058 1797 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal uuid: "21544559ee284906bc8977db1465db7b" format_stamp: "Formatted at 2019-07-20 00:42:34 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:36.361989 1797 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal 4 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Did not check for full LBM containers with extra space Did not check for incomplete LBM containers Did not check for malformed LBM records Did not check for misaligned LBM blocks Did not check for partial LBM records I0720 00:42:36.390748 1797 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:36.392467 1797 ts_tablet_manager.cc:345] Loading tablet metadata (0/4 complete) I0720 00:42:36.400557 1797 ts_tablet_manager.cc:357] Loaded tablet metadata (4 total tablets, 4 live tablets) I0720 00:42:36.400625 1797 ts_tablet_manager.cc:363] Registering tablets (0/4 complete) I0720 00:42:36.403479 1821 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Bootstrap starting. E0720 00:42:36.403780 1821 ts_tablet_manager.cc:1080] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:36.403856 1821 ts_tablet_manager.cc:1059] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0720 00:42:36.403956 1821 tablet_replica.cc:273] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: stopping tablet replica I0720 00:42:36.404067 1821 raft_consensus.cc:2147] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:36.404932 1821 raft_consensus.cc:2174] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:36.406826 1821 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Bootstrap starting. E0720 00:42:36.406978 1821 ts_tablet_manager.cc:1080] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:36.407060 1821 ts_tablet_manager.cc:1059] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:42:36.407160 1821 tablet_replica.cc:273] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: stopping tablet replica I0720 00:42:36.407263 1821 raft_consensus.cc:2147] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:36.407331 1821 raft_consensus.cc:2174] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:36.409175 1821 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:36.411214 1797 ts_tablet_manager.cc:377] Registered 4 tablets I0720 00:42:36.411630 1822 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Bootstrap starting. E0720 00:42:36.411830 1822 ts_tablet_manager.cc:1080] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:36.412176 1822 ts_tablet_manager.cc:1059] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0720 00:42:36.412253 1822 tablet_replica.cc:273] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: stopping tablet replica I0720 00:42:36.412326 1822 raft_consensus.cc:2147] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:36.412420 1822 raft_consensus.cc:2174] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:36.424166 1821 log.cc:525] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Log is configured to *not* fsync() on all Append() calls I0720 00:42:36.433185 1780 consensus_queue.cc:984] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:36.434584 1821 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates W0720 00:42:36.434690 1604 consensus_peers.cc:458] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:33685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:36.435631 1821 tablet_bootstrap.cc:439] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Bootstrap complete. I0720 00:42:36.436578 1771 consensus_queue.cc:984] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:36.437078 1821 ts_tablet_manager.cc:1059] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.015s W0720 00:42:36.438597 1478 consensus_peers.cc:458] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:33685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:42:36.438750 1478 consensus_peers.cc:458] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:33685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:36.454509 1821 raft_consensus.cc:340] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:36.458001 1821 raft_consensus.cc:705] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: FOLLOWER I0720 00:42:36.458672 1821 consensus_queue.cc:229] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } } I0720 00:42:36.460420 1821 ts_tablet_manager.cc:1087] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.023s user 0.018s sys 0.003s I0720 00:42:36.462826 1797 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:33685 I0720 00:42:36.464041 1797 webserver.cc:374] Webserver started at http://127.0.58.129:38721/ using document root and password file I0720 00:42:36.464741 1797 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:42:36.466403 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 1797 W0720 00:42:36.468336 1933 tablet.cc:1799] T 8b450357aeba4ea4b2b282750617b863 P 21544559ee284906bc8977db1465db7b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:36.475189 1932 heartbeater.cc:324] Connected to a master server at 127.0.58.190:33793 I0720 00:42:36.475376 1932 heartbeater.cc:405] Registering TS with master... I0720 00:42:36.475812 1932 heartbeater.cc:454] Master 127.0.58.190:33793 requested a full tablet report, sending... I0720 00:42:36.477879 1167 ts_manager.cc:161] Re-registered known tserver with Master: 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) W0720 00:42:36.478343 1167 catalog_manager.cc:3988] Tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) has failed on TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): IO error: some tablet data is in a failed directory W0720 00:42:36.478471 1167 catalog_manager.cc:3988] Tablet c362ba31e9ec40999c7c048ce93e99ff (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) has failed on TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): IO error: some tablet data is in a failed directory W0720 00:42:36.478552 1167 catalog_manager.cc:3988] Tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) has failed on TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): IO error: some tablet data is in a failed directory I0720 00:42:36.479848 1167 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:51643 I0720 00:42:36.513765 1352 consensus_queue.cc:544] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: The tablet replica hosted on peer 21544559ee284906bc8977db1465db7b has failed W0720 00:42:36.513999 1352 consensus_peers.cc:458] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:36.535456 1779 consensus_queue.cc:984] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- cfcd88079c52477ebcae1d98c42dcb23 | 127.0.58.190:33793 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | HEALTHY | 21544559ee284906bc8977db1465db7b | 127.0.58.129:33685 | HEALTHY | da17ba98538b41b091e9cbf779bc5afa | 127.0.58.131:39787 | HEALTHY | f554abf9579c42029052fc648fb4bd8c | 127.0.58.130:45963 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:33685 env_inject_eio | 1 | hidden | 127.0.58.129:33685 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:33685 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:33685 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:45963 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:39787 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:39821 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:33685 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:45963 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:39787 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:39821 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 7a6d7c8f8b6340fab210da8b3907aa6e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING [LEADER] 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory All reported replicas are: A = da17ba98538b41b091e9cbf779bc5afa B = f554abf9579c42029052fc648fb4bd8c C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | [config not available] | | | Tablet c362ba31e9ec40999c7c048ce93e99ff of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821): RUNNING [LEADER] All reported replicas are: A = 21544559ee284906bc8977db1465db7b B = f554abf9579c42029052fc648fb4bd8c C = 0eeb775c53b34d00ab90623d4b54577f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 8b450357aeba4ea4b2b282750617b863 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING [LEADER] 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821): RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): RUNNING All reported replicas are: A = da17ba98538b41b091e9cbf779bc5afa B = 0eeb775c53b34d00ab90623d4b54577f C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet 5aa1a45e61fa4786a8f6e49ac00c36fd of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING [LEADER] All reported replicas are: A = 21544559ee284906bc8977db1465db7b B = f554abf9579c42029052fc648fb4bd8c C = da17ba98538b41b091e9cbf779bc5afa The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 1 | 0 | 3 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:36.558324 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:36.606530 1801 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:36.615895 1167 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 7a6d7c8f8b6340fab210da8b3907aa6e (attempt 1) I0720 00:42:36.617717 1418 consensus_queue.cc:206] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } I0720 00:42:36.619457 1545 raft_consensus.cc:1184] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Refusing update from remote peer f554abf9579c42029052fc648fb4bd8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:36.619947 1801 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:36.622054 1772 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } } I0720 00:42:36.622624 1545 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } } I0720 00:42:36.710191 1150 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 7a6d7c8f8b6340fab210da8b3907aa6e with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:42:36.711088 1166 catalog_manager.cc:4067] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c reported cstate change: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } } I0720 00:42:36.715543 1166 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7a6d7c8f8b6340fab210da8b3907aa6e on 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 2) I0720 00:42:36.715898 1166 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7a6d7c8f8b6340fab210da8b3907aa6e (attempt 1) I0720 00:42:36.717731 1418 consensus_queue.cc:206] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:36.720397 1352 consensus_queue.cc:544] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: The tablet replica hosted on peer 21544559ee284906bc8977db1465db7b has failed W0720 00:42:36.720513 1352 consensus_peers.cc:458] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:36.721264 1545 raft_consensus.cc:1184] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Refusing update from remote peer f554abf9579c42029052fc648fb4bd8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:42:36.721741 1770 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:42:36.724023 1868 tablet_service.cc:1066] Processing DeleteTablet for tablet 7a6d7c8f8b6340fab210da8b3907aa6e with delete_type TABLET_DATA_TOMBSTONED (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:32820 I0720 00:42:36.724040 1772 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:36.724509 1544 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:36.726542 1977 ts_tablet_manager.cc:1400] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:36.814908 1150 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7a6d7c8f8b6340fab210da8b3907aa6e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:42:36.815827 1166 catalog_manager.cc:4067] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa reported cstate change: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:36.816148 1977 ts_tablet_manager.cc:1413] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:36.816267 1977 log.cc:990] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal/wals/7a6d7c8f8b6340fab210da8b3907aa6e I0720 00:42:36.817420 1151 catalog_manager.cc:3437] TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) successfully deleted I0720 00:42:36.819259 1167 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 7a6d7c8f8b6340fab210da8b3907aa6e (attempt 1) I0720 00:42:36.820749 1418 consensus_queue.cc:206] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } I0720 00:42:36.822170 1545 raft_consensus.cc:1184] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Refusing update from remote peer f554abf9579c42029052fc648fb4bd8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:36.822563 1770 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- cfcd88079c52477ebcae1d98c42dcb23 | 127.0.58.190:33793 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | HEALTHY | 21544559ee284906bc8977db1465db7b | 127.0.58.129:33685 | HEALTHY | da17ba98538b41b091e9cbf779bc5afa | 127.0.58.131:39787 | HEALTHY | f554abf9579c42029052fc648fb4bd8c | 127.0.58.130:45963 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:33685 env_inject_eio | 1 | hidden | 127.0.58.129:33685 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:33685 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:33685 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:45963 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:39787 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:39821 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:33685 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:45963 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:39787 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:39821 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 7a6d7c8f8b6340fab210da8b3907aa6e of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING [LEADER] I0720 00:42:36.824568 1801 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } } All reported replicas are: A = da17ba98538b41b091e9cbf779bc5afa B = f554abf9579c42029052fc648fb4bd8c C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes Tablet c362ba31e9ec40999c7c048ce93e99ff of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821): RUNNING [LEADER] All reported replicas are: A = 21544559ee284906bc8977db1465db7b B = f554abf9579c42029052fc648fb4bd8c C = 0eeb775c53b34d00ab90623d4b54577f I0720 00:42:36.825040 1545 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } } The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 8b450357aeba4ea4b2b282750617b863 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING [LEADER] 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821): RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): RUNNING All reported replicas are: A = da17ba98538b41b091e9cbf779bc5afa B = 0eeb775c53b34d00ab90623d4b54577f C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | A* B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet 5aa1a45e61fa4786a8f6e49ac00c36fd of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING [LEADER] All reported replicas are: A = 21544559ee284906bc8977db1465db7b B = f554abf9579c42029052fc648fb4bd8c C = da17ba98538b41b091e9cbf779bc5afa The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 1 | 0 | 2 | 2 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 3 Median | 4 Third Quartile | 4 Maximum | 4 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 14 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:36.829809 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:36.914993 1150 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 7a6d7c8f8b6340fab210da8b3907aa6e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:42:36.916518 1167 catalog_manager.cc:4067] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c reported cstate change: config changed from index 3 to 4, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } } I0720 00:42:36.919487 1167 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7a6d7c8f8b6340fab210da8b3907aa6e on 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 4) I0720 00:42:36.919651 1167 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7a6d7c8f8b6340fab210da8b3907aa6e (attempt 1) I0720 00:42:36.920251 1868 tablet_service.cc:1066] Processing DeleteTablet for tablet 7a6d7c8f8b6340fab210da8b3907aa6e with delete_type TABLET_DATA_TOMBSTONED (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:32820 I0720 00:42:36.920809 1151 catalog_manager.cc:3437] TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) successfully deleted I0720 00:42:36.920914 1418 consensus_queue.cc:206] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:36.922646 1545 raft_consensus.cc:1184] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Refusing update from remote peer f554abf9579c42029052fc648fb4bd8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) W0720 00:42:36.922749 1352 consensus_peers.cc:458] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:36.923271 1996 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0720 00:42:36.925369 1772 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:36.925935 1545 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:37.001274 1478 consensus_queue.cc:544] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: The tablet replica hosted on peer 21544559ee284906bc8977db1465db7b has failed I0720 00:42:37.003476 1167 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (attempt 1) I0720 00:42:37.005033 1544 consensus_queue.cc:206] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } I0720 00:42:37.006788 1418 raft_consensus.cc:1184] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Refusing update from remote peer da17ba98538b41b091e9cbf779bc5afa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:37.007369 1779 consensus_queue.cc:984] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:37.009783 1782 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } } I0720 00:42:37.010507 1418 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } } I0720 00:42:37.014153 1604 consensus_queue.cc:544] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: The tablet replica hosted on peer 21544559ee284906bc8977db1465db7b has failed I0720 00:42:37.018442 1166 catalog_manager.cc:4067] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa reported cstate change: config changed from index 4 to 5, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:37.019870 1151 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:42:37.020040 1150 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7a6d7c8f8b6340fab210da8b3907aa6e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:42:37.020766 1167 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet c362ba31e9ec40999c7c048ce93e99ff (attempt 1) I0720 00:42:37.022477 1671 consensus_queue.cc:206] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } I0720 00:42:37.022886 1166 catalog_manager.cc:4067] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa reported cstate change: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "da17ba98538b41b091e9cbf779bc5afa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } } I0720 00:42:37.024329 1418 raft_consensus.cc:1184] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Refusing update from remote peer 0eeb775c53b34d00ab90623d4b54577f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:37.024804 1771 consensus_queue.cc:984] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:37.025617 1166 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd on 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 2) I0720 00:42:37.026208 1868 tablet_service.cc:1066] Processing DeleteTablet for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd with delete_type TABLET_DATA_TOMBSTONED (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:32820 I0720 00:42:37.026470 1977 ts_tablet_manager.cc:1400] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:37.026832 1999 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } } I0720 00:42:37.027287 1418 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } } I0720 00:42:37.028429 1166 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (attempt 1) I0720 00:42:37.029469 1545 consensus_queue.cc:206] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.031214 1419 raft_consensus.cc:1184] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Refusing update from remote peer da17ba98538b41b091e9cbf779bc5afa: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:42:37.031679 1780 consensus_queue.cc:984] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s W0720 00:42:37.031904 1478 consensus_peers.cc:458] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:37.033465 1779 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:37.033946 1419 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:37.037693 1932 heartbeater.cc:446] Master 127.0.58.190:33793 was elected leader, sending a full tablet report... W0720 00:42:37.041076 1166 catalog_manager.cc:3988] Tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) has failed on TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): IO error: some tablet data is in a failed directory W0720 00:42:37.041165 1166 catalog_manager.cc:3988] Tablet c362ba31e9ec40999c7c048ce93e99ff (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) has failed on TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): IO error: some tablet data is in a failed directory W0720 00:42:37.041213 1166 catalog_manager.cc:3988] Tablet 7a6d7c8f8b6340fab210da8b3907aa6e (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) has failed on TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): IO error: some tablet data is in a failed directory I0720 00:42:37.114053 1151 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:42:37.115808 1166 catalog_manager.cc:4067] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa reported cstate change: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "da17ba98538b41b091e9cbf779bc5afa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:42:37.119364 1152 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet c362ba31e9ec40999c7c048ce93e99ff with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:42:37.120700 1166 catalog_manager.cc:4067] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f reported cstate change: config changed from index -1 to 2, VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "0eeb775c53b34d00ab90623d4b54577f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } health_report { overall_health: HEALTHY } } } I0720 00:42:37.126531 1977 ts_tablet_manager.cc:1413] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:37.126626 1977 log.cc:990] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal/wals/5aa1a45e61fa4786a8f6e49ac00c36fd I0720 00:42:37.127307 1151 catalog_manager.cc:3437] TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): tablet 5aa1a45e61fa4786a8f6e49ac00c36fd (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) successfully deleted I0720 00:42:37.127676 1166 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet c362ba31e9ec40999c7c048ce93e99ff on 21544559ee284906bc8977db1465db7b (127.0.58.129:33685) (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 2) I0720 00:42:37.127902 1166 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c362ba31e9ec40999c7c048ce93e99ff (attempt 1) I0720 00:42:37.128290 1868 tablet_service.cc:1066] Processing DeleteTablet for tablet c362ba31e9ec40999c7c048ce93e99ff with delete_type TABLET_DATA_TOMBSTONED (TS 21544559ee284906bc8977db1465db7b not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:32820 I0720 00:42:37.128577 1977 ts_tablet_manager.cc:1400] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:37.129264 1671 consensus_queue.cc:206] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } W0720 00:42:37.131223 1604 consensus_peers.cc:458] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:37.131286 1418 raft_consensus.cc:1184] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Refusing update from remote peer 0eeb775c53b34d00ab90623d4b54577f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:42:37.131821 1999 consensus_queue.cc:984] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:42:37.134093 1771 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:37.134708 1418 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } } I0720 00:42:37.204452 1152 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c362ba31e9ec40999c7c048ce93e99ff with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:42:37.209112 1168 catalog_manager.cc:4067] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f reported cstate change: config changed from index 2 to 3, NON_VOTER 21544559ee284906bc8977db1465db7b (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "0eeb775c53b34d00ab90623d4b54577f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- cfcd88079c52477ebcae1d98c42dcb23 | 127.0.58.190:33793 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | HEALTHY | 21544559ee284906bc8977db1465db7b | 127.0.58.129:33685 | HEALTHY | da17ba98538b41b091e9cbf779bc5afa | 127.0.58.131:39787 | HEALTHY | f554abf9579c42029052fc648fb4bd8c | 127.0.58.130:45963 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:33685 env_inject_eio | 1 | hidden | 127.0.58.129:33685 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:33685 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:33685 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:45963 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:39787 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:39821 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:33685 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:45963 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:39787 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:39821 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet c362ba31e9ec40999c7c048ce93e99ff of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821): RUNNING [LEADER] All reported replicas are: A = 21544559ee284906bc8977db1465db7b B = f554abf9579c42029052fc648fb4bd8c C = 0eeb775c53b34d00ab90623d4b54577f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A~ B C* | 1 | 3 | Yes C | B C* | 1 | 2 | Yes Tablet 5aa1a45e61fa4786a8f6e49ac00c36fd of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING [LEADER] All reported replicas are: A = f554abf9579c42029052fc648fb4bd8c B = da17ba98538b41b091e9cbf779bc5afa C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 1 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 3 Median | 4 Third Quartile | 4 Maximum | 4 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 14 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:37.218910 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:37.300470 1977 ts_tablet_manager.cc:1413] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:37.300613 1977 log.cc:990] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/wal/wals/c362ba31e9ec40999c7c048ce93e99ff I0720 00:42:37.301424 1151 catalog_manager.cc:3437] TS 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): tablet c362ba31e9ec40999c7c048ce93e99ff (table test-workload [id=3f4ea59900aa4bb38ec9fca128138efc]) successfully deleted I0720 00:42:37.382025 2027 ts_tablet_manager.cc:682] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Initiating tablet copy from peer f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963) I0720 00:42:37.382288 2027 tablet_copy_client.cc:204] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:37.382442 2027 tablet_copy_client.cc:241] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.130:45963 I0720 00:42:37.388316 1439 tablet_copy_service.cc:135] P f554abf9579c42029052fc648fb4bd8c: Received BeginTabletCopySession request for tablet 7a6d7c8f8b6340fab210da8b3907aa6e from peer 21544559ee284906bc8977db1465db7b ({username='slave'} at 127.0.58.129:54031) I0720 00:42:37.388489 1439 tablet_copy_service.cc:156] P f554abf9579c42029052fc648fb4bd8c: Beginning new tablet copy session on tablet 7a6d7c8f8b6340fab210da8b3907aa6e from peer 21544559ee284906bc8977db1465db7b at {username='slave'} at 127.0.58.129:54031: session id = 21544559ee284906bc8977db1465db7b-7a6d7c8f8b6340fab210da8b3907aa6e I0720 00:42:37.390736 1439 tablet_copy_source_session.cc:202] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:37.392258 2027 ts_tablet_manager.cc:1400] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:42:37.398007 2027 ts_tablet_manager.cc:1413] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:37.497530 2027 tablet_copy_client.cc:579] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet copy: Starting download of 0 data blocks... I0720 00:42:37.497798 2027 tablet_copy_client.cc:542] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet copy: Starting download of 1 WAL segments... I0720 00:42:37.500296 2027 tablet_copy_client.cc:418] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:37.513345 2029 ts_tablet_manager.cc:682] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Initiating tablet copy from peer da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787) I0720 00:42:37.513444 2029 tablet_copy_client.cc:204] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:37.513617 2029 tablet_copy_client.cc:241] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.131:39787 I0720 00:42:37.518967 1565 tablet_copy_service.cc:135] P da17ba98538b41b091e9cbf779bc5afa: Received BeginTabletCopySession request for tablet 5aa1a45e61fa4786a8f6e49ac00c36fd from peer 21544559ee284906bc8977db1465db7b ({username='slave'} at 127.0.58.129:55609) I0720 00:42:37.519145 1565 tablet_copy_service.cc:156] P da17ba98538b41b091e9cbf779bc5afa: Beginning new tablet copy session on tablet 5aa1a45e61fa4786a8f6e49ac00c36fd from peer 21544559ee284906bc8977db1465db7b at {username='slave'} at 127.0.58.129:55609: session id = 21544559ee284906bc8977db1465db7b-5aa1a45e61fa4786a8f6e49ac00c36fd I0720 00:42:37.521258 1565 tablet_copy_source_session.cc:202] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:37.522475 2029 ts_tablet_manager.cc:1400] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:42:37.563751 2034 ts_tablet_manager.cc:682] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Initiating tablet copy from peer 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821) I0720 00:42:37.563840 2034 tablet_copy_client.cc:204] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:37.563946 2034 tablet_copy_client.cc:241] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.132:39821 I0720 00:42:37.570004 1691 tablet_copy_service.cc:135] P 0eeb775c53b34d00ab90623d4b54577f: Received BeginTabletCopySession request for tablet c362ba31e9ec40999c7c048ce93e99ff from peer 21544559ee284906bc8977db1465db7b ({username='slave'} at 127.0.58.129:45703) I0720 00:42:37.570204 1691 tablet_copy_service.cc:156] P 0eeb775c53b34d00ab90623d4b54577f: Beginning new tablet copy session on tablet c362ba31e9ec40999c7c048ce93e99ff from peer 21544559ee284906bc8977db1465db7b at {username='slave'} at 127.0.58.129:45703: session id = 21544559ee284906bc8977db1465db7b-c362ba31e9ec40999c7c048ce93e99ff I0720 00:42:37.572321 1691 tablet_copy_source_session.cc:202] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:37.573500 2034 ts_tablet_manager.cc:1400] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:42:37.601354 2027 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:37.611369 2027 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:37.611851 2027 tablet_bootstrap.cc:439] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Bootstrap complete. I0720 00:42:37.612057 2027 ts_tablet_manager.cc:1059] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.001s I0720 00:42:37.612835 2027 raft_consensus.cc:340] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.613019 2027 raft_consensus.cc:705] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: LEARNER I0720 00:42:37.613183 2027 consensus_queue.cc:229] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.614835 2027 ts_tablet_manager.cc:1087] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I0720 00:42:37.617050 1439 tablet_copy_service.cc:325] P f554abf9579c42029052fc648fb4bd8c: Request end of tablet copy session 21544559ee284906bc8977db1465db7b-7a6d7c8f8b6340fab210da8b3907aa6e received from {username='slave'} at 127.0.58.129:54031 I0720 00:42:37.617166 1439 tablet_copy_service.cc:413] P f554abf9579c42029052fc648fb4bd8c: ending tablet copy session 21544559ee284906bc8977db1465db7b-7a6d7c8f8b6340fab210da8b3907aa6e on tablet 7a6d7c8f8b6340fab210da8b3907aa6e with peer 21544559ee284906bc8977db1465db7b Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- cfcd88079c52477ebcae1d98c42dcb23 | 127.0.58.190:33793 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | HEALTHY | 21544559ee284906bc8977db1465db7b | 127.0.58.129:33685 | HEALTHY | da17ba98538b41b091e9cbf779bc5afa | 127.0.58.131:39787 | HEALTHY | f554abf9579c42029052fc648fb4bd8c | 127.0.58.130:45963 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:33685 env_inject_eio | 1 | hidden | 127.0.58.129:33685 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:33685 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:33685 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:45963 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:39787 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:39821 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:33685 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:45963 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:39787 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:39821 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 7a6d7c8f8b6340fab210da8b3907aa6e of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING [LEADER] 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): RUNNING [NONVOTER] All reported replicas are: A = da17ba98538b41b091e9cbf779bc5afa B = f554abf9579c42029052fc648fb4bd8c C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 5 | Yes B | A B* C~ | 1 | 5 | Yes C | A B C~ | 1 | 5 | Yes Tablet c362ba31e9ec40999c7c048ce93e99ff of table 'test-workload' is recovering: 1 on-going tablet copies f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING 0eeb775c53b34d00ab90623d4b54577f (127.0.58.132:39821): RUNNING [LEADER] 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running [NONVOTER] State: FAILED Data state: TABLET_DATA_COPYING Last status: Deleted tablet blocks from disk All reported replicas are: A = f554abf9579c42029052fc648fb4bd8c B = 0eeb775c53b34d00ab90623d4b54577f C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes C | [config not available] | | | Tablet 5aa1a45e61fa4786a8f6e49ac00c36fd of table 'test-workload' is recovering: 1 on-going tablet copies f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): RUNNING da17ba98538b41b091e9cbf779bc5afa (127.0.58.131:39787): RUNNING [LEADER] 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): not running [NONVOTER] State: FAILED Data state: TABLET_DATA_COPYING Last status: Deleted tablet blocks from disk All reported replicas are: A = f554abf9579c42029052fc648fb4bd8c B = da17ba98538b41b091e9cbf779bc5afa C = 21544559ee284906bc8977db1465db7b The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 2 | 2 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:37.639080 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:37.701041 2029 ts_tablet_manager.cc:1413] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:37.703068 2034 ts_tablet_manager.cc:1413] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:37.704458 2029 tablet_copy_client.cc:579] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet copy: Starting download of 0 data blocks... I0720 00:42:37.704927 2029 tablet_copy_client.cc:542] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet copy: Starting download of 1 WAL segments... I0720 00:42:37.706831 2029 tablet_copy_client.cc:418] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:37.776604 1888 raft_consensus.cc:1126] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[] I0720 00:42:37.802637 2034 tablet_copy_client.cc:579] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet copy: Starting download of 0 data blocks... I0720 00:42:37.802848 2034 tablet_copy_client.cc:542] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet copy: Starting download of 1 WAL segments... I0720 00:42:37.802901 2029 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:37.804926 2034 tablet_copy_client.cc:418] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:37.807672 2034 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Bootstrap starting. I0720 00:42:37.809794 2029 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:37.810379 2029 tablet_bootstrap.cc:439] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Bootstrap complete. I0720 00:42:37.810577 2029 ts_tablet_manager.cc:1059] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0720 00:42:37.811319 2029 raft_consensus.cc:340] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.811542 2029 raft_consensus.cc:705] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: LEARNER I0720 00:42:37.811724 2029 consensus_queue.cc:229] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.812263 2029 ts_tablet_manager.cc:1087] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0720 00:42:37.813103 1565 tablet_copy_service.cc:325] P da17ba98538b41b091e9cbf779bc5afa: Request end of tablet copy session 21544559ee284906bc8977db1465db7b-5aa1a45e61fa4786a8f6e49ac00c36fd received from {username='slave'} at 127.0.58.129:55609 I0720 00:42:37.813231 1565 tablet_copy_service.cc:413] P da17ba98538b41b091e9cbf779bc5afa: ending tablet copy session 21544559ee284906bc8977db1465db7b-5aa1a45e61fa4786a8f6e49ac00c36fd on tablet 5aa1a45e61fa4786a8f6e49ac00c36fd with peer 21544559ee284906bc8977db1465db7b I0720 00:42:37.814599 2034 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:37.815011 2034 tablet_bootstrap.cc:439] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Bootstrap complete. I0720 00:42:37.815196 2034 ts_tablet_manager.cc:1059] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s I0720 00:42:37.815794 2034 raft_consensus.cc:340] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.815951 2034 raft_consensus.cc:705] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21544559ee284906bc8977db1465db7b, State: Initialized, Role: LEARNER I0720 00:42:37.816082 2034 consensus_queue.cc:229] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: true } } I0720 00:42:37.816501 2034 ts_tablet_manager.cc:1087] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0720 00:42:37.817193 1691 tablet_copy_service.cc:325] P 0eeb775c53b34d00ab90623d4b54577f: Request end of tablet copy session 21544559ee284906bc8977db1465db7b-c362ba31e9ec40999c7c048ce93e99ff received from {username='slave'} at 127.0.58.129:45703 I0720 00:42:37.817324 1691 tablet_copy_service.cc:413] P 0eeb775c53b34d00ab90623d4b54577f: ending tablet copy session 21544559ee284906bc8977db1465db7b-c362ba31e9ec40999c7c048ce93e99ff on tablet c362ba31e9ec40999c7c048ce93e99ff with peer 21544559ee284906bc8977db1465db7b I0720 00:42:38.003542 1888 raft_consensus.cc:1126] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[] I0720 00:42:38.050931 1888 raft_consensus.cc:1126] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[] Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- cfcd88079c52477ebcae1d98c42dcb23 | 127.0.58.190:33793 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | HEALTHY | 21544559ee284906bc8977db1465db7b | 127.0.58.129:33685 | HEALTHY | da17ba98538b41b091e9cbf779bc5afa | 127.0.58.131:39787 | HEALTHY | f554abf9579c42029052fc648fb4bd8c | 127.0.58.130:45963 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:33685 env_inject_eio | 1 | hidden | 127.0.58.129:33685 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:33685 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:33685 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:45963 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:39787 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:39821 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:33685 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:45963 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:39787 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:39821 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 0eeb775c53b34d00ab90623d4b54577f | 127.0.58.132:39821 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:42:38.229027 234 log_verifier.cc:121] Checking tablet 1aa1fb28c3d546e88c2d549fe18c1b76 I0720 00:42:38.248178 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 1aa1fb28c3d546e88c2d549fe18c1b76 I0720 00:42:38.248234 234 log_verifier.cc:121] Checking tablet 5aa1a45e61fa4786a8f6e49ac00c36fd I0720 00:42:38.315161 234 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 5aa1a45e61fa4786a8f6e49ac00c36fd I0720 00:42:38.315228 234 log_verifier.cc:121] Checking tablet 7a6d7c8f8b6340fab210da8b3907aa6e I0720 00:42:38.335780 234 log_verifier.cc:172] Verified matching terms for 5 ops in tablet 7a6d7c8f8b6340fab210da8b3907aa6e I0720 00:42:38.335855 234 log_verifier.cc:121] Checking tablet 8b450357aeba4ea4b2b282750617b863 I0720 00:42:38.357712 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 8b450357aeba4ea4b2b282750617b863 I0720 00:42:38.357767 234 log_verifier.cc:121] Checking tablet c362ba31e9ec40999c7c048ce93e99ff I0720 00:42:38.369429 2058 raft_consensus.cc:974] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c: attempting to promote NON_VOTER 21544559ee284906bc8977db1465db7b to VOTER I0720 00:42:38.370180 2058 consensus_queue.cc:206] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } I0720 00:42:38.374125 1888 raft_consensus.cc:1184] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Refusing update from remote peer f554abf9579c42029052fc648fb4bd8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I0720 00:42:38.375036 2058 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0720 00:42:38.376356 1545 raft_consensus.cc:1184] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Refusing update from remote peer f554abf9579c42029052fc648fb4bd8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I0720 00:42:38.376994 2093 consensus_queue.cc:984] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0720 00:42:38.379462 2058 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.380216 1545 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P da17ba98538b41b091e9cbf779bc5afa [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.380025 1888 raft_consensus.cc:2792] T 7a6d7c8f8b6340fab210da8b3907aa6e P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.387607 234 log_verifier.cc:172] Verified matching terms for 3 ops in tablet c362ba31e9ec40999c7c048ce93e99ff I0720 00:42:38.407359 1166 catalog_manager.cc:4067] T 7a6d7c8f8b6340fab210da8b3907aa6e P f554abf9579c42029052fc648fb4bd8c reported cstate change: config changed from index 5 to 6, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f554abf9579c42029052fc648fb4bd8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0720 00:42:38.537531 2064 raft_consensus.cc:974] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f: attempting to promote NON_VOTER 21544559ee284906bc8977db1465db7b to VOTER I0720 00:42:38.538126 2064 consensus_queue.cc:206] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } I0720 00:42:38.540882 1888 raft_consensus.cc:1184] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Refusing update from remote peer 0eeb775c53b34d00ab90623d4b54577f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:38.540910 1418 raft_consensus.cc:1184] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Refusing update from remote peer 0eeb775c53b34d00ab90623d4b54577f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:38.541462 2063 consensus_queue.cc:984] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:38.541807 1999 consensus_queue.cc:984] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:38.544484 1999 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.545143 1418 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.545732 1888 raft_consensus.cc:2792] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.611737 2062 raft_consensus.cc:974] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa: attempting to promote NON_VOTER 21544559ee284906bc8977db1465db7b to VOTER I0720 00:42:38.612362 2062 consensus_queue.cc:206] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } I0720 00:42:38.614329 1887 raft_consensus.cc:1184] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 LEARNER]: Refusing update from remote peer da17ba98538b41b091e9cbf779bc5afa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:38.615187 2113 consensus_queue.cc:984] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:38.616552 1417 raft_consensus.cc:1184] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Refusing update from remote peer da17ba98538b41b091e9cbf779bc5afa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:38.617086 2111 consensus_queue.cc:984] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:38.623353 2113 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.624048 1417 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P f554abf9579c42029052fc648fb4bd8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.624106 1887 raft_consensus.cc:2792] T 5aa1a45e61fa4786a8f6e49ac00c36fd P 21544559ee284906bc8977db1465db7b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.703284 1167 catalog_manager.cc:4067] T c362ba31e9ec40999c7c048ce93e99ff P 21544559ee284906bc8977db1465db7b reported cstate change: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0eeb775c53b34d00ab90623d4b54577f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } } peers { permanent_uuid: "0eeb775c53b34d00ab90623d4b54577f" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 39821 } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } } } I0720 00:42:38.706724 1165 catalog_manager.cc:4067] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa reported cstate change: config changed from index 3 to 4, 21544559ee284906bc8977db1465db7b (127.0.58.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "da17ba98538b41b091e9cbf779bc5afa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f554abf9579c42029052fc648fb4bd8c" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 45963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da17ba98538b41b091e9cbf779bc5afa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 39787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21544559ee284906bc8977db1465db7b" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 33685 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0720 00:42:38.781626 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 1797 I0720 00:42:38.795251 1929 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:38.873438 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 1797 I0720 00:42:38.888424 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 1339 I0720 00:42:38.894834 1459 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:42:38.926687 1604 consensus_peers.cc:458] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:33685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:42:38.966728 1478 consensus_peers.cc:458] T 8b450357aeba4ea4b2b282750617b863 P da17ba98538b41b091e9cbf779bc5afa -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:33685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:38.986665 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 1339 I0720 00:42:39.000955 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 1465 I0720 00:42:39.008080 1585 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:42:39.030849 1603 consensus_peers.cc:458] T c362ba31e9ec40999c7c048ce93e99ff P 0eeb775c53b34d00ab90623d4b54577f -> Peer f554abf9579c42029052fc648fb4bd8c (127.0.58.130:45963): Couldn't send request to peer f554abf9579c42029052fc648fb4bd8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.130:45963: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:42:39.115577 1478 consensus_peers.cc:458] T 5aa1a45e61fa4786a8f6e49ac00c36fd P da17ba98538b41b091e9cbf779bc5afa -> Peer 21544559ee284906bc8977db1465db7b (127.0.58.129:33685): Couldn't send request to peer 21544559ee284906bc8977db1465db7b. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:33685: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:39.115758 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 1465 I0720 00:42:39.128446 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 1591 I0720 00:42:39.134290 1711 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:39.214540 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 1591 I0720 00:42:39.228330 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 1138 I0720 00:42:39.229375 1198 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:39.310897 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 1138 [ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/1 (5513 ms) [ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2 I0720 00:42:39.669268 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-4 --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:43971 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:39.880494 2132 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:39.880736 2132 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:39.883365 2132 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:39.883422 2132 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:42:39.883455 2132 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:39.883483 2132 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:42:39.883527 2132 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:42:39.888463 2132 master_main.cc:98] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:43971 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:39.895728 2132 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 41016us I0720 00:42:39.897270 2132 fs_manager.cc:263] Metadata directory not provided I0720 00:42:39.897341 2132 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:39.897706 2132 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:39.897778 2132 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:39.900580 2132 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.901123 2132 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.901492 2132 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.901798 2132 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-3/instance: uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.902110 2132 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-4/instance: uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.902426 2132 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.914196 2132 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.008s sys 0.004s I0720 00:42:39.915763 2132 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:39.916363 2132 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:42:39.920346 2132 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0720 00:42:39.920462 2132 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/wal uuid: "65741a0735184fc583c84c30265e2e4e" format_stamp: "Formatted at 2019-07-20 00:42:39 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:39.920657 2132 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:42:40.078850 2132 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:40.092543 2132 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:43971 I0720 00:42:40.093780 2132 webserver.cc:374] Webserver started at http://127.0.58.190:35341/ using document root and password file I0720 00:42:40.094558 2132 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:42:40.096123 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 2132 I0720 00:42:40.096942 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:43971 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:40.112148 2195 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e: Bootstrap starting. I0720 00:42:40.115164 2195 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e: No blocks or log segments found. Creating new log. I0720 00:42:40.116050 2195 log.cc:525] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e: Log is configured to *not* fsync() on all Append() calls I0720 00:42:40.117585 2195 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e: No bootstrap required, opened a new log I0720 00:42:40.124006 2195 raft_consensus.cc:340] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } I0720 00:42:40.124274 2195 raft_consensus.cc:366] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:40.124370 2195 raft_consensus.cc:705] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65741a0735184fc583c84c30265e2e4e, State: Initialized, Role: FOLLOWER I0720 00:42:40.124927 2195 consensus_queue.cc:229] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } I0720 00:42:40.125221 2195 raft_consensus.cc:380] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:42:40.125294 2195 raft_consensus.cc:472] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:42:40.125433 2195 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 0 FOLLOWER]: Advancing to term 1 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:40.269906 2196 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:40.270176 2196 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:40.272269 2196 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:40.272326 2196 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:40.272485 2196 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:42:40.276744 2196 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:43971 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:40.285097 2196 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 41516us I0720 00:42:40.285925 2196 fs_manager.cc:263] Metadata directory not provided I0720 00:42:40.286152 2196 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:40.286553 2196 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:40.286628 2196 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:40.289762 2196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.290343 2196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.290710 2196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.291031 2196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-3/instance: uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.291383 2196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-4/instance: uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.291743 2196 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.303772 2196 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.006s sys 0.005s I0720 00:42:40.305299 2196 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:40.305883 2196 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:42:40.309237 2196 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:42:40.309324 2196 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/wal uuid: "d9d5e137df31450caa9dc831972c1f5e" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.309481 2196 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:42:40.325500 2196 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:40.327230 2196 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:40.327307 2196 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:40.356603 2196 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:37035 I0720 00:42:40.357700 2196 webserver.cc:374] Webserver started at http://127.0.58.129:38925/ using document root and password file I0720 00:42:40.358359 2196 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:42:40.367180 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 2196 I0720 00:42:40.367921 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:43971 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:40.381378 2320 heartbeater.cc:324] Connected to a master server at 127.0.58.190:43971 I0720 00:42:40.381603 2320 heartbeater.cc:405] Registering TS with master... I0720 00:42:40.382107 2320 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... W0720 00:42:40.384436 2320 heartbeater.cc:570] Failed to heartbeat to 127.0.58.190:43971 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting I0720 00:42:40.384644 2320 heartbeater.cc:405] Registering TS with master... I0720 00:42:40.384857 2320 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... I0720 00:42:40.386129 2320 heartbeater.cc:405] Registering TS with master... I0720 00:42:40.386292 2320 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... W0720 00:42:40.387270 2320 heartbeater.cc:359] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts. I0720 00:42:40.422592 2195 raft_consensus.cc:494] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } I0720 00:42:40.423398 2195 leader_election.cc:310] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65741a0735184fc583c84c30265e2e4e; no voters: I0720 00:42:40.424505 2195 leader_election.cc:296] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:42:40.424777 2325 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:40.426229 2325 raft_consensus.cc:667] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [term 1 LEADER]: Becoming Leader. State: Replica: 65741a0735184fc583c84c30265e2e4e, State: Running, Role: LEADER I0720 00:42:40.426681 2325 consensus_queue.cc:206] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } I0720 00:42:40.430589 2195 sys_catalog.cc:340] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "65741a0735184fc583c84c30265e2e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } } I0720 00:42:40.430780 2326 sys_catalog.cc:340] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65741a0735184fc583c84c30265e2e4e. Latest consensus state: current_term: 1 leader_uuid: "65741a0735184fc583c84c30265e2e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } } I0720 00:42:40.431015 2326 sys_catalog.cc:343] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: This master's current role is: LEADER I0720 00:42:40.430923 2195 sys_catalog.cc:343] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: This master's current role is: LEADER I0720 00:42:40.430753 2327 sys_catalog.cc:340] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65741a0735184fc583c84c30265e2e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65741a0735184fc583c84c30265e2e4e" member_type: VOTER } } I0720 00:42:40.431188 2327 sys_catalog.cc:343] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: This master's current role is: LEADER I0720 00:42:40.432588 2195 sys_catalog.cc:428] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e [sys.catalog]: configured and running, proceeding with master startup. I0720 00:42:40.432767 2330 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:42:40.433082 2331 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0720 00:42:40.440382 2330 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:42:40.467159 2330 catalog_manager.cc:959] Generated new certificate authority record I0720 00:42:40.468582 2330 catalog_manager.cc:1084] Loading token signing keys... I0720 00:42:40.479286 2330 catalog_manager.cc:4371] T 00000000000000000000000000000000 P 65741a0735184fc583c84c30265e2e4e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:40.524318 2324 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:40.524539 2324 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:40.526612 2324 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:40.526669 2324 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:40.526777 2324 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:42:40.542536 2324 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:43971 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:40.550276 2324 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 41516us I0720 00:42:40.551026 2324 fs_manager.cc:263] Metadata directory not provided I0720 00:42:40.551087 2324 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:40.551395 2324 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:40.551460 2324 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:40.553692 2324 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.554280 2324 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.554625 2324 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.554916 2324 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3/instance: uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.555218 2324 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4/instance: uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.555521 2324 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.570324 2324 fs_manager.cc:503] Time spent creating directory manager: real 0.015s user 0.011s sys 0.005s I0720 00:42:40.571897 2324 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:40.572410 2324 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:42:40.576212 2324 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s I0720 00:42:40.576299 2324 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.576463 2324 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:42:40.611140 2324 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:40.612790 2324 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:40.612875 2324 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:40.642819 2324 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:37359 I0720 00:42:40.643954 2324 webserver.cc:374] Webserver started at http://127.0.58.130:37945/ using document root and password file I0720 00:42:40.644584 2324 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:42:40.653666 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 2324 I0720 00:42:40.654292 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:43971 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:40.665138 2456 heartbeater.cc:324] Connected to a master server at 127.0.58.190:43971 I0720 00:42:40.665405 2456 heartbeater.cc:405] Registering TS with master... I0720 00:42:40.666003 2456 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... I0720 00:42:40.667600 2162 ts_manager.cc:161] Registered new tserver with Master: bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) I0720 00:42:40.669296 2162 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:41745 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:40.828796 2459 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:40.829051 2459 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:40.831216 2459 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:40.831281 2459 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:40.831390 2459 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:42:40.835817 2459 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:43971 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:40.845028 2459 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 41516us I0720 00:42:40.845896 2459 fs_manager.cc:263] Metadata directory not provided I0720 00:42:40.845969 2459 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:42:40.846431 2459 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:42:40.846506 2459 server_base.cc:449] Attempting to create new FS layout instead I0720 00:42:40.849107 2459 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.849674 2459 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.850004 2459 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.850333 2459 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-3/instance: uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.850666 2459 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-4/instance: uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.850962 2459 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.863276 2459 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.010s sys 0.001s I0720 00:42:40.865670 2459 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:40.866520 2459 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:42:40.871024 2459 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0720 00:42:40.871116 2459 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/wal uuid: "526eadb7abb3438790a38e0d9973a6a5" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:40.871356 2459 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:42:40.907321 2459 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:40.908934 2459 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:42:40.909005 2459 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:42:40.933688 2459 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:33941 I0720 00:42:40.934857 2459 webserver.cc:374] Webserver started at http://127.0.58.131:33675/ using document root and password file I0720 00:42:40.935535 2459 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:42:40.940619 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 2459 I0720 00:42:40.956377 2582 heartbeater.cc:324] Connected to a master server at 127.0.58.190:43971 I0720 00:42:40.956619 2582 heartbeater.cc:405] Registering TS with master... I0720 00:42:40.957099 2582 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... I0720 00:42:40.958233 2162 ts_manager.cc:161] Registered new tserver with Master: 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:40.959102 2162 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:37465 I0720 00:42:41.387457 2320 heartbeater.cc:405] Registering TS with master... I0720 00:42:41.387858 2320 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... I0720 00:42:41.388880 2161 ts_manager.cc:161] Registered new tserver with Master: d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:41.389889 2161 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:44599 I0720 00:42:41.391252 234 external_mini_cluster.cc:628] 3 TS(s) registered with all masters I0720 00:42:41.401468 234 test_util.cc:212] Using random seed: 517292641 I0720 00:42:41.416491 2161 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43754: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff" } partition_schema { range_schema { columns { name: "key" } } } I0720 00:42:41.448756 2390 tablet_service.cc:1022] Processing CreateTablet for tablet ade4b6ce17504569a6d13d9018288194 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:41.448967 2257 tablet_service.cc:1022] Processing CreateTablet for tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:41.450206 2391 tablet_service.cc:1022] Processing CreateTablet for tablet 7e3e1eff26974b46b75eac1cc0290d3b (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:41.450639 2392 tablet_service.cc:1022] Processing CreateTablet for tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:41.448679 2389 tablet_service.cc:1022] Processing CreateTablet for tablet fc71eee696004a62810a0f8ac4b1185d (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:41.450445 2518 tablet_service.cc:1022] Processing CreateTablet for tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:41.450405 2255 tablet_service.cc:1022] Processing CreateTablet for tablet 7e3e1eff26974b46b75eac1cc0290d3b (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:41.454447 2517 tablet_service.cc:1022] Processing CreateTablet for tablet 7e3e1eff26974b46b75eac1cc0290d3b (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:42:41.450853 2254 tablet_service.cc:1022] Processing CreateTablet for tablet ade4b6ce17504569a6d13d9018288194 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:41.451303 2253 tablet_service.cc:1022] Processing CreateTablet for tablet fc71eee696004a62810a0f8ac4b1185d (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:41.455483 2516 tablet_service.cc:1022] Processing CreateTablet for tablet ade4b6ce17504569a6d13d9018288194 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:42:41.451619 2256 tablet_service.cc:1022] Processing CreateTablet for tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:42:41.456423 2515 tablet_service.cc:1022] Processing CreateTablet for tablet fc71eee696004a62810a0f8ac4b1185d (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:42:41.449035 2393 tablet_service.cc:1022] Processing CreateTablet for tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:41.459488 2519 tablet_service.cc:1022] Processing CreateTablet for tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (table=test-workload [id=4c1b1a3e42344401926664c866bb23b0]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:42:41.483120 2606 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:41.484966 2608 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:41.488085 2607 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:41.491891 2610 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Bootstrap starting. I0720 00:42:41.492431 2608 tablet_bootstrap.cc:590] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: No blocks or log segments found. Creating new log. I0720 00:42:41.492962 2609 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:41.494055 2607 tablet_bootstrap.cc:590] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: No blocks or log segments found. Creating new log. I0720 00:42:41.496839 2606 tablet_bootstrap.cc:590] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: No blocks or log segments found. Creating new log. I0720 00:42:41.498131 2606 log.cc:525] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Log is configured to *not* fsync() on all Append() calls I0720 00:42:41.498677 2616 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Bootstrap starting. I0720 00:42:41.499775 2609 tablet_bootstrap.cc:590] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: No blocks or log segments found. Creating new log. I0720 00:42:41.500919 2611 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:41.499218 2614 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Bootstrap starting. I0720 00:42:41.503331 2606 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: No bootstrap required, opened a new log I0720 00:42:41.503517 2619 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5: Bootstrap starting. I0720 00:42:41.504650 2607 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: No bootstrap required, opened a new log I0720 00:42:41.506480 2606 ts_tablet_manager.cc:1059] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s I0720 00:42:41.506701 2607 ts_tablet_manager.cc:1059] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s I0720 00:42:41.508034 2615 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: Bootstrap starting. I0720 00:42:41.508579 2612 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e: Bootstrap starting. I0720 00:42:41.510282 2611 tablet_bootstrap.cc:590] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: No blocks or log segments found. Creating new log. I0720 00:42:41.510557 2623 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Bootstrap starting. I0720 00:42:41.510529 2609 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: No bootstrap required, opened a new log I0720 00:42:41.511250 2621 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e: Bootstrap starting. I0720 00:42:41.511373 2609 ts_tablet_manager.cc:1059] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s I0720 00:42:41.512135 2614 tablet_bootstrap.cc:590] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: No blocks or log segments found. Creating new log. I0720 00:42:41.512715 2616 tablet_bootstrap.cc:590] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: No blocks or log segments found. Creating new log. I0720 00:42:41.512889 2610 tablet_bootstrap.cc:590] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: No blocks or log segments found. Creating new log. I0720 00:42:41.513528 2624 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Bootstrap starting. I0720 00:42:41.513809 2619 tablet_bootstrap.cc:590] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5: No blocks or log segments found. Creating new log. I0720 00:42:41.513985 2608 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: No bootstrap required, opened a new log I0720 00:42:41.514155 2608 ts_tablet_manager.cc:1059] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s I0720 00:42:41.515264 2612 tablet_bootstrap.cc:590] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e: No blocks or log segments found. Creating new log. I0720 00:42:41.515579 2615 tablet_bootstrap.cc:590] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: No blocks or log segments found. Creating new log. I0720 00:42:41.517225 2614 log.cc:525] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Log is configured to *not* fsync() on all Append() calls I0720 00:42:41.517501 2610 log.cc:525] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Log is configured to *not* fsync() on all Append() calls I0720 00:42:41.518277 2623 tablet_bootstrap.cc:590] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: No blocks or log segments found. Creating new log. I0720 00:42:41.518723 2621 tablet_bootstrap.cc:590] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e: No blocks or log segments found. Creating new log. I0720 00:42:41.519317 2624 tablet_bootstrap.cc:590] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: No blocks or log segments found. Creating new log. I0720 00:42:41.523221 2615 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: No bootstrap required, opened a new log I0720 00:42:41.523533 2615 ts_tablet_manager.cc:1059] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s I0720 00:42:41.523567 2613 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: Bootstrap starting. I0720 00:42:41.525550 2621 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e: No bootstrap required, opened a new log I0720 00:42:41.525655 2621 ts_tablet_manager.cc:1059] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s I0720 00:42:41.527446 2624 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: No bootstrap required, opened a new log I0720 00:42:41.527585 2624 ts_tablet_manager.cc:1059] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s I0720 00:42:41.528437 2614 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: No bootstrap required, opened a new log I0720 00:42:41.528600 2623 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: No bootstrap required, opened a new log I0720 00:42:41.528717 2612 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e: No bootstrap required, opened a new log I0720 00:42:41.528452 2619 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5: No bootstrap required, opened a new log I0720 00:42:41.528815 2614 ts_tablet_manager.cc:1059] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s I0720 00:42:41.528851 2612 ts_tablet_manager.cc:1059] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.005s I0720 00:42:41.528898 2619 ts_tablet_manager.cc:1059] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s I0720 00:42:41.529086 2623 ts_tablet_manager.cc:1059] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s I0720 00:42:41.530475 2610 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: No bootstrap required, opened a new log I0720 00:42:41.530616 2610 ts_tablet_manager.cc:1059] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.005s I0720 00:42:41.531652 2611 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: No bootstrap required, opened a new log I0720 00:42:41.531854 2611 ts_tablet_manager.cc:1059] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.003s I0720 00:42:41.533985 2616 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: No bootstrap required, opened a new log I0720 00:42:41.534199 2616 ts_tablet_manager.cc:1059] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.002s I0720 00:42:41.534955 2613 tablet_bootstrap.cc:590] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: No blocks or log segments found. Creating new log. I0720 00:42:41.547192 2613 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: No bootstrap required, opened a new log I0720 00:42:41.547407 2613 ts_tablet_manager.cc:1059] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s I0720 00:42:41.554209 2623 raft_consensus.cc:340] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.555238 2623 raft_consensus.cc:366] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.555372 2623 raft_consensus.cc:705] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526eadb7abb3438790a38e0d9973a6a5, State: Initialized, Role: FOLLOWER I0720 00:42:41.554976 2614 raft_consensus.cc:340] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.555922 2614 raft_consensus.cc:366] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.556002 2614 raft_consensus.cc:705] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526eadb7abb3438790a38e0d9973a6a5, State: Initialized, Role: FOLLOWER I0720 00:42:41.556536 2614 consensus_queue.cc:229] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.556942 2623 consensus_queue.cc:229] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.554029 2613 raft_consensus.cc:340] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.564134 2611 raft_consensus.cc:340] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.566315 2611 raft_consensus.cc:366] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.566699 2611 raft_consensus.cc:705] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:41.567268 2611 consensus_queue.cc:229] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.566485 2606 raft_consensus.cc:340] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.569267 2606 raft_consensus.cc:366] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.569346 2606 raft_consensus.cc:705] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:41.566540 2608 raft_consensus.cc:340] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.568549 2456 heartbeater.cc:446] Master 127.0.58.190:43971 was elected leader, sending a full tablet report... I0720 00:42:41.569903 2608 raft_consensus.cc:366] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.569998 2608 raft_consensus.cc:705] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:41.570190 2608 consensus_queue.cc:229] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.570566 2608 ts_tablet_manager.cc:1087] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.056s user 0.006s sys 0.004s I0720 00:42:41.566622 2607 raft_consensus.cc:340] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.570847 2607 raft_consensus.cc:366] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.571010 2607 raft_consensus.cc:705] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:41.571213 2607 consensus_queue.cc:229] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.566407 2609 raft_consensus.cc:340] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.571677 2609 raft_consensus.cc:366] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.564924 2615 raft_consensus.cc:340] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.565055 2610 raft_consensus.cc:340] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.569190 2611 ts_tablet_manager.cc:1087] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.037s user 0.010s sys 0.024s I0720 00:42:41.573232 2610 raft_consensus.cc:366] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.573365 2610 raft_consensus.cc:705] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Initialized, Role: FOLLOWER I0720 00:42:41.573248 2615 raft_consensus.cc:366] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.565093 2612 raft_consensus.cc:340] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.574228 2612 raft_consensus.cc:366] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.565093 2624 raft_consensus.cc:340] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.574355 2624 raft_consensus.cc:366] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.572645 2621 raft_consensus.cc:340] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.574443 2621 raft_consensus.cc:366] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.574496 2610 consensus_queue.cc:229] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.575007 2612 raft_consensus.cc:705] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Initialized, Role: FOLLOWER I0720 00:42:41.575379 2612 consensus_queue.cc:229] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.575659 2613 raft_consensus.cc:366] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.575819 2624 raft_consensus.cc:705] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Initialized, Role: FOLLOWER I0720 00:42:41.575785 2616 raft_consensus.cc:340] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.576078 2616 raft_consensus.cc:366] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.576045 2624 consensus_queue.cc:229] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.576371 2320 heartbeater.cc:446] Master 127.0.58.190:43971 was elected leader, sending a full tablet report... I0720 00:42:41.576153 2619 raft_consensus.cc:340] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.576839 2619 raft_consensus.cc:366] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:42:41.576915 2619 raft_consensus.cc:705] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526eadb7abb3438790a38e0d9973a6a5, State: Initialized, Role: FOLLOWER I0720 00:42:41.576933 2624 ts_tablet_manager.cc:1087] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Time spent starting tablet: real 0.049s user 0.006s sys 0.013s I0720 00:42:41.577118 2619 consensus_queue.cc:229] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.577198 2621 raft_consensus.cc:705] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Initialized, Role: FOLLOWER I0720 00:42:41.577484 2613 raft_consensus.cc:705] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526eadb7abb3438790a38e0d9973a6a5, State: Initialized, Role: FOLLOWER I0720 00:42:41.577400 2621 consensus_queue.cc:229] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.577841 2621 ts_tablet_manager.cc:1087] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e: Time spent starting tablet: real 0.052s user 0.000s sys 0.006s I0720 00:42:41.577955 2612 ts_tablet_manager.cc:1087] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e: Time spent starting tablet: real 0.049s user 0.004s sys 0.012s I0720 00:42:41.578097 2615 raft_consensus.cc:705] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Initialized, Role: FOLLOWER I0720 00:42:41.577517 2616 raft_consensus.cc:705] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526eadb7abb3438790a38e0d9973a6a5, State: Initialized, Role: FOLLOWER I0720 00:42:41.578390 2615 consensus_queue.cc:229] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.578508 2613 consensus_queue.cc:229] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.578577 2616 consensus_queue.cc:229] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.586506 2615 ts_tablet_manager.cc:1087] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: Time spent starting tablet: real 0.063s user 0.004s sys 0.004s I0720 00:42:41.587333 2582 heartbeater.cc:446] Master 127.0.58.190:43971 was elected leader, sending a full tablet report... I0720 00:42:41.588042 2609 raft_consensus.cc:705] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:41.588088 2616 ts_tablet_manager.cc:1087] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Time spent starting tablet: real 0.054s user 0.008s sys 0.006s I0720 00:42:41.588315 2609 consensus_queue.cc:229] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.588481 2623 ts_tablet_manager.cc:1087] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Time spent starting tablet: real 0.059s user 0.006s sys 0.003s I0720 00:42:41.588722 2614 ts_tablet_manager.cc:1087] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Time spent starting tablet: real 0.060s user 0.000s sys 0.009s I0720 00:42:41.588932 2613 ts_tablet_manager.cc:1087] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s I0720 00:42:41.589115 2619 ts_tablet_manager.cc:1087] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5: Time spent starting tablet: real 0.060s user 0.004s sys 0.007s I0720 00:42:41.589359 2606 consensus_queue.cc:229] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.589821 2610 ts_tablet_manager.cc:1087] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Time spent starting tablet: real 0.059s user 0.014s sys 0.017s I0720 00:42:41.589442 2609 ts_tablet_manager.cc:1087] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.077s user 0.000s sys 0.009s I0720 00:42:41.590397 2606 ts_tablet_manager.cc:1087] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.084s user 0.007s sys 0.000s I0720 00:42:41.590828 2607 ts_tablet_manager.cc:1087] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.084s user 0.000s sys 0.010s W0720 00:42:41.610129 2321 tablet.cc:1799] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:42:41.647125 2457 tablet.cc:1799] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:42:41.687603 2583 tablet.cc:1799] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:41.708361 2640 raft_consensus.cc:472] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:41.708626 2640 raft_consensus.cc:494] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.710053 2640 leader_election.cc:296] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) I0720 00:42:41.713979 2640 raft_consensus.cc:472] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:41.714259 2640 raft_consensus.cc:494] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.714958 2640 leader_election.cc:296] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:41.717321 2539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fc71eee696004a62810a0f8ac4b1185d" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:41.717315 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "18b33e9892f14d0abfa3d7ecb1ee9e80" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:41.717871 2539 raft_consensus.cc:2350] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.717876 2538 raft_consensus.cc:2350] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.718160 2413 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fc71eee696004a62810a0f8ac4b1185d" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true I0720 00:42:41.718544 2413 raft_consensus.cc:2350] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.718572 2209 leader_election.cc:310] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:41.718989 2209 leader_election.cc:310] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:41.718906 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "18b33e9892f14d0abfa3d7ecb1ee9e80" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true I0720 00:42:41.719027 2640 raft_consensus.cc:2644] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:41.719107 2640 raft_consensus.cc:472] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:41.719138 2412 raft_consensus.cc:2350] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.719221 2640 raft_consensus.cc:2890] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:41.719424 2641 raft_consensus.cc:2644] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:41.719529 2641 raft_consensus.cc:472] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:41.719616 2641 raft_consensus.cc:2890] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:41.788393 2639 raft_consensus.cc:472] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:41.788624 2639 raft_consensus.cc:494] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.790035 2639 leader_election.cc:296] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:41.798027 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:41.798310 2538 raft_consensus.cc:2350] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc957945380a4e119d7ac829148eede6 in term 0. I0720 00:42:41.798149 2277 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" is_pre_election: true I0720 00:42:41.798507 2277 raft_consensus.cc:2350] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc957945380a4e119d7ac829148eede6 in term 0. I0720 00:42:41.798820 2345 leader_election.cc:310] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, bc957945380a4e119d7ac829148eede6; no voters: I0720 00:42:41.799223 2639 raft_consensus.cc:2644] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:41.799311 2639 raft_consensus.cc:472] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:41.799409 2639 raft_consensus.cc:2890] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:41.825587 2649 raft_consensus.cc:472] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:41.825783 2649 raft_consensus.cc:494] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.826541 2649 leader_election.cc:296] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:41.827297 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ade4b6ce17504569a6d13d9018288194" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true I0720 00:42:41.827389 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ade4b6ce17504569a6d13d9018288194" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:41.827476 2412 raft_consensus.cc:2350] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.827636 2538 raft_consensus.cc:2350] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.827956 2209 leader_election.cc:310] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc957945380a4e119d7ac829148eede6, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:41.828243 2649 raft_consensus.cc:2644] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:41.828335 2649 raft_consensus.cc:472] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:41.828402 2649 raft_consensus.cc:2890] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:41.961277 2650 raft_consensus.cc:472] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:41.961500 2650 raft_consensus.cc:494] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:41.962404 2650 leader_election.cc:296] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:41.963204 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true I0720 00:42:41.963238 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:41.963400 2412 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.963418 2538 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 0. I0720 00:42:41.963776 2209 leader_election.cc:310] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc957945380a4e119d7ac829148eede6, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:41.964176 2650 raft_consensus.cc:2644] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:41.964277 2650 raft_consensus.cc:472] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:41.964344 2650 raft_consensus.cc:2890] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.062805 2640 raft_consensus.cc:494] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:42.062813 2639 raft_consensus.cc:494] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:42.062988 2641 raft_consensus.cc:494] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:42.063530 2640 leader_election.cc:296] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Requested vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) I0720 00:42:42.063812 2639 leader_election.cc:296] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 election: Requested vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:42.064221 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:42.064534 2537 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "18b33e9892f14d0abfa3d7ecb1ee9e80" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:42.064604 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fc71eee696004a62810a0f8ac4b1185d" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" I0720 00:42:42.064653 2537 raft_consensus.cc:2890] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.064762 2412 raft_consensus.cc:2890] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.064837 2413 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "18b33e9892f14d0abfa3d7ecb1ee9e80" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" I0720 00:42:42.064983 2413 raft_consensus.cc:2890] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.065443 2539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "fc71eee696004a62810a0f8ac4b1185d" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:42.065601 2539 raft_consensus.cc:2890] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.066784 2537 raft_consensus.cc:2350] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:42.066993 2538 raft_consensus.cc:2890] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.067761 2641 leader_election.cc:296] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Requested vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:42.068498 2277 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" I0720 00:42:42.068666 2209 leader_election.cc:310] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:42.068715 2277 raft_consensus.cc:2890] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.069052 2641 raft_consensus.cc:2644] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:42.069229 2641 raft_consensus.cc:667] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Becoming Leader. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Running, Role: LEADER I0720 00:42:42.069638 2641 consensus_queue.cc:206] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:42.073173 2161 catalog_manager.cc:4067] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e reported cstate change: term changed from 0 to 1, leader changed from to d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:42.468996 2641 consensus_queue.cc:984] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0720 00:42:42.889176 2539 consensus_meta.cc:220] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.823s user 0.002s sys 0.000s W0720 00:42:42.889176 2649 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 1.061s user 0.002s sys 0.000s W0720 00:42:42.889176 2413 consensus_meta.cc:220] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Time spent flushing consensus metadata: real 0.824s user 0.002s sys 0.000s I0720 00:42:42.889302 2539 raft_consensus.cc:2350] T fc71eee696004a62810a0f8ac4b1185d P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:42.889299 2413 raft_consensus.cc:2350] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:42.889315 2649 raft_consensus.cc:494] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } W0720 00:42:42.889829 2412 consensus_meta.cc:220] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Time spent flushing consensus metadata: real 0.825s user 0.002s sys 0.000s I0720 00:42:42.889902 2412 raft_consensus.cc:2350] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. W0720 00:42:42.890008 2538 consensus_meta.cc:220] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.823s user 0.001s sys 0.000s I0720 00:42:42.890059 2538 raft_consensus.cc:2350] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc957945380a4e119d7ac829148eede6 in term 1. I0720 00:42:42.890069 2649 leader_election.cc:296] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Requested vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:42.890475 2209 leader_election.cc:310] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:42.890810 2649 raft_consensus.cc:2644] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:42.890960 2649 raft_consensus.cc:667] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Becoming Leader. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Running, Role: LEADER I0720 00:42:42.891171 2649 consensus_queue.cc:206] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:42.891384 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ade4b6ce17504569a6d13d9018288194" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" I0720 00:42:42.891609 2412 raft_consensus.cc:2890] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.892418 2640 consensus_queue.cc:984] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:42.892644 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ade4b6ce17504569a6d13d9018288194" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:42.892863 2538 raft_consensus.cc:2890] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:42:42.893406 2345 leader_election.cc:310] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, bc957945380a4e119d7ac829148eede6; no voters: I0720 00:42:42.898887 2660 raft_consensus.cc:2644] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:42.899636 2660 raft_consensus.cc:667] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 1 LEADER]: Becoming Leader. State: Replica: bc957945380a4e119d7ac829148eede6, State: Running, Role: LEADER I0720 00:42:42.900070 2660 consensus_queue.cc:206] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:42.903875 2161 catalog_manager.cc:4067] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e reported cstate change: term changed from 0 to 1, leader changed from to d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:42.905215 2162 catalog_manager.cc:4067] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 reported cstate change: term changed from 0 to 1, leader changed from to bc957945380a4e119d7ac829148eede6 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "bc957945380a4e119d7ac829148eede6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:43.272693 2649 consensus_queue.cc:984] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:43.277393 2649 consensus_queue.cc:984] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:43.514716 2660 consensus_queue.cc:984] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:43.521914 2660 raft_consensus.cc:472] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:43.522037 2660 raft_consensus.cc:494] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:43.522614 2660 leader_election.cc:296] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:43.523046 2539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:43.523205 2539 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc957945380a4e119d7ac829148eede6 in term 0. I0720 00:42:43.523252 2275 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" is_pre_election: true I0720 00:42:43.523556 2345 leader_election.cc:310] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, bc957945380a4e119d7ac829148eede6; no voters: I0720 00:42:43.523905 2660 raft_consensus.cc:2644] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:42:43.524003 2660 raft_consensus.cc:472] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:43.524071 2660 raft_consensus.cc:2890] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 0 FOLLOWER]: Advancing to term 1 W0720 00:42:43.690239 2347 leader_election.cc:341] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): Timed out: RequestConsensusVote RPC to 127.0.58.129:37035 timed out after 1.625s (SENT) W0720 00:42:43.716946 2650 consensus_meta.cc:220] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 1.752s user 0.002s sys 0.000s W0720 00:42:43.717134 2412 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent flushing consensus metadata: real 0.825s user 0.002s sys 0.000s I0720 00:42:43.717062 2650 raft_consensus.cc:494] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:43.717221 2412 raft_consensus.cc:2350] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:43.717775 2650 leader_election.cc:296] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Requested vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:43.718037 2275 raft_consensus.cc:2280] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bc957945380a4e119d7ac829148eede6 in current term 1: Already voted for candidate d9d5e137df31450caa9dc831972c1f5e in this term. I0720 00:42:43.718724 2209 leader_election.cc:310] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc957945380a4e119d7ac829148eede6, d9d5e137df31450caa9dc831972c1f5e; no voters: W0720 00:42:43.718981 2277 consensus_meta.cc:220] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 1.650s user 0.002s sys 0.000s I0720 00:42:43.719055 2277 raft_consensus.cc:2350] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc957945380a4e119d7ac829148eede6 in term 1. I0720 00:42:43.719128 2650 raft_consensus.cc:2644] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:42:43.719257 2650 raft_consensus.cc:667] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Becoming Leader. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Running, Role: LEADER I0720 00:42:43.719467 2650 consensus_queue.cc:206] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:43.719916 2539 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:43.720142 2539 raft_consensus.cc:2890] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 0 FOLLOWER]: Advancing to term 1 W0720 00:42:43.719189 2277 rpcz_store.cc:253] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.58.130:51597 (request call id 2) took 1654 ms (1.65 s). Client timeout 1625 ms (1.62 s) I0720 00:42:43.721303 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" W0720 00:42:43.721899 2277 rpcz_store.cc:259] Trace: 0720 00:42:42.064507 (+ 0us) service_pool.cc:162] Inserting onto call queue 0720 00:42:42.064561 (+ 54us) service_pool.cc:221] Handling call 0720 00:42:43.719130 (+1654569us) inbound_call.cc:162] Queueing success response Metrics: {} I0720 00:42:43.722043 2660 raft_consensus.cc:494] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:43.722249 2675 consensus_queue.cc:984] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0720 00:42:43.722440 2412 raft_consensus.cc:2280] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9d5e137df31450caa9dc831972c1f5e in current term 1: Already voted for candidate bc957945380a4e119d7ac829148eede6 in this term. I0720 00:42:43.723057 2660 leader_election.cc:296] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 election: Requested vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:43.722901 2162 catalog_manager.cc:4067] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e reported cstate change: term changed from 0 to 1, leader changed from to d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:43.723295 2277 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" I0720 00:42:43.723320 2537 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "bc957945380a4e119d7ac829148eede6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:43.723465 2277 raft_consensus.cc:2280] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc957945380a4e119d7ac829148eede6 in current term 1: Already voted for candidate d9d5e137df31450caa9dc831972c1f5e in this term. W0720 00:42:43.745707 2538 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.853s user 0.002s sys 0.000s I0720 00:42:43.745815 2538 raft_consensus.cc:2350] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:43.983813 2650 raft_consensus.cc:472] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:42:43.983990 2650 raft_consensus.cc:494] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:43.984634 2650 leader_election.cc:296] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:43.985213 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true I0720 00:42:43.985285 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:43.985399 2412 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:43.985819 2209 leader_election.cc:310] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc957945380a4e119d7ac829148eede6, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:43.986063 2650 raft_consensus.cc:2644] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:42:43.986157 2650 raft_consensus.cc:472] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:42:43.986202 2650 raft_consensus.cc:2890] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:44.108347 2649 consensus_queue.cc:984] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:42:44.113922 2683 consensus_queue.cc:984] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0720 00:42:44.453011 2539 consensus_meta.cc:220] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.733s user 0.002s sys 0.000s I0720 00:42:44.453145 2539 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 1. I0720 00:42:44.453050 2650 raft_consensus.cc:494] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:44.453418 2537 raft_consensus.cc:2325] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc957945380a4e119d7ac829148eede6 for term 1 because replica is already servicing an update from a current leader or another vote. I0720 00:42:44.453791 2650 leader_election.cc:296] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 election: Requested vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941) I0720 00:42:44.453888 2538 raft_consensus.cc:2325] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9d5e137df31450caa9dc831972c1f5e for term 2 because replica is already servicing an update from a current leader or another vote. I0720 00:42:44.454250 2345 leader_election.cc:310] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bc957945380a4e119d7ac829148eede6; no voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e I0720 00:42:44.454331 2209 leader_election.cc:310] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: bc957945380a4e119d7ac829148eede6 I0720 00:42:44.454607 2650 raft_consensus.cc:2607] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Leader election decision vote started in defunct term 1: won I0720 00:42:44.454610 2675 raft_consensus.cc:2592] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority I0720 00:42:44.455204 2412 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" I0720 00:42:44.455377 2412 raft_consensus.cc:2890] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:44.455417 2538 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7e3e1eff26974b46b75eac1cc0290d3b" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" I0720 00:42:44.455586 2538 raft_consensus.cc:2890] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Advancing to term 2 W0720 00:42:45.253862 2538 consensus_meta.cc:220] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.798s user 0.002s sys 0.000s W0720 00:42:45.253881 2412 consensus_meta.cc:220] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Time spent flushing consensus metadata: real 0.798s user 0.002s sys 0.000s I0720 00:42:45.254012 2412 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 2. I0720 00:42:45.254001 2538 raft_consensus.cc:2350] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9d5e137df31450caa9dc831972c1f5e in term 2. I0720 00:42:45.254680 2209 leader_election.cc:310] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc957945380a4e119d7ac829148eede6, d9d5e137df31450caa9dc831972c1f5e; no voters: I0720 00:42:45.255074 2650 raft_consensus.cc:2644] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:42:45.255461 2650 raft_consensus.cc:667] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [term 2 LEADER]: Becoming Leader. State: Replica: d9d5e137df31450caa9dc831972c1f5e, State: Running, Role: LEADER I0720 00:42:45.255996 2650 consensus_queue.cc:206] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:45.258383 2162 catalog_manager.cc:4067] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e reported cstate change: term changed from 0 to 2, leader changed from to d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 2 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:45.525336 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 2324 I0720 00:42:45.525935 2453 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:42:45.581109 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 2324 I0720 00:42:45.592417 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.130:37359 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_port=37945 --webserver_interface=127.0.58.130 --tserver_master_addrs=127.0.58.190:43971 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** --env_inject_eio=1.0 --crash_on_eio=false --raft_prepare_replacement_before_eviction=false with env {} I0720 00:42:45.635732 2538 raft_consensus.cc:1184] T 7e3e1eff26974b46b75eac1cc0290d3b P 526eadb7abb3438790a38e0d9973a6a5 [term 2 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0720 00:42:45.636289 2687 consensus_queue.cc:984] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0720 00:42:45.645948 2209 consensus_peers.cc:458] T 7e3e1eff26974b46b75eac1cc0290d3b P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.130:37359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:42:45.657147 2209 consensus_peers.cc:458] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.130:37359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:42:45.711294 2209 consensus_peers.cc:458] T fc71eee696004a62810a0f8ac4b1185d P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.130:37359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:42:45.754506 2700 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:42:45.754766 2700 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:42:45.756840 2700 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0720 00:42:45.756906 2700 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:42:45.757005 2700 flags.cc:404] Enabled experimental flag: --crash_on_eio=false W0720 00:42:45.757079 2700 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:42:45.761354 2700 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:37359 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=37945 --tserver_master_addrs=127.0.58.190:43971 --env_inject_eio=1 --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:42:45.769459 2700 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 44016us W0720 00:42:45.769747 2700 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:42:45.769793 2700 fault_injection.cc:45] THIS SERVER MAY CRASH! I0720 00:42:45.770406 2700 fs_manager.cc:263] Metadata directory not provided I0720 00:42:45.770516 2700 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory E0720 00:42:45.771876 2700 env_posix.cc:316] I/O error, context: INJECTED FAILURE E0720 00:42:45.773702 2700 env_posix.cc:316] I/O error, context: INJECTED FAILURE I0720 00:42:45.773772 2700 block_manager_util.cc:139] Instance is unhealthy: IO error: Failed to read metadata file from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/data/block_manager_instance: INJECTED FAILURE: Input/output error (error 5) W0720 00:42:45.778236 2209 consensus_peers.cc:458] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.130:37359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:42:45.780175 2700 fs_manager.cc:399] Time spent opening directory manager: real 0.007s user 0.007s sys 0.001s I0720 00:42:45.781251 2700 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:42:45.781782 2700 file_cache.cc:465] Constructed file cache lbm with capacity 419430 E0720 00:42:45.784806 2700 log_block_manager.cc:2011] Not using report from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/data: IO error: Data directory failed (error 5) I0720 00:42:45.785010 2700 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0720 00:42:45.785074 2700 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal uuid: "bc957945380a4e119d7ac829148eede6" format_stamp: "Formatted at 2019-07-20 00:42:40 on dist-test-slave-dist-test-slave-lkfw" I0720 00:42:45.786063 2700 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal 4 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:42:45.807142 2700 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:42:45.808766 2700 ts_tablet_manager.cc:345] Loading tablet metadata (0/5 complete) I0720 00:42:45.818614 2700 ts_tablet_manager.cc:357] Loaded tablet metadata (5 total tablets, 5 live tablets) I0720 00:42:45.818691 2700 ts_tablet_manager.cc:363] Registering tablets (0/5 complete) I0720 00:42:45.821710 2723 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. E0720 00:42:45.822041 2723 ts_tablet_manager.cc:1080] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:45.822165 2723 ts_tablet_manager.cc:1059] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0720 00:42:45.822288 2723 tablet_replica.cc:273] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: stopping tablet replica I0720 00:42:45.822432 2723 raft_consensus.cc:2147] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:45.823786 2724 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:45.823797 2723 raft_consensus.cc:2174] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:45.825736 2723 tablet_bootstrap.cc:439] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. E0720 00:42:45.825887 2723 ts_tablet_manager.cc:1080] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:45.825966 2723 ts_tablet_manager.cc:1059] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:42:45.826032 2723 tablet_replica.cc:273] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: stopping tablet replica I0720 00:42:45.826141 2723 raft_consensus.cc:2147] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:45.826207 2723 raft_consensus.cc:2174] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:45.830265 2723 tablet_bootstrap.cc:439] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. E0720 00:42:45.830428 2723 ts_tablet_manager.cc:1080] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:42:45.830487 2700 ts_tablet_manager.cc:377] Registered 5 tablets I0720 00:42:45.830588 2723 ts_tablet_manager.cc:1059] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:42:45.830670 2723 tablet_replica.cc:273] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6: stopping tablet replica I0720 00:42:45.830858 2723 raft_consensus.cc:2147] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:42:45.830904 2725 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:45.830963 2723 raft_consensus.cc:2174] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:42:45.842288 2724 log.cc:525] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Log is configured to *not* fsync() on all Append() calls I0720 00:42:45.846477 2725 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:45.847198 2725 tablet_bootstrap.cc:439] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Bootstrap complete. I0720 00:42:45.848033 2725 ts_tablet_manager.cc:1059] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s I0720 00:42:45.848095 2724 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:45.848637 2724 tablet_bootstrap.cc:439] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Bootstrap complete. I0720 00:42:45.849989 2724 ts_tablet_manager.cc:1059] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.016s I0720 00:42:45.868640 2724 raft_consensus.cc:340] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:45.868067 2725 raft_consensus.cc:340] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:45.871763 2724 raft_consensus.cc:705] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:45.871758 2725 raft_consensus.cc:705] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:45.872501 2724 consensus_queue.cc:229] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:45.872504 2725 consensus_queue.cc:229] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:45.874526 2725 ts_tablet_manager.cc:1087] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.026s user 0.014s sys 0.006s I0720 00:42:45.874521 2724 ts_tablet_manager.cc:1087] T fc71eee696004a62810a0f8ac4b1185d P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.024s user 0.013s sys 0.009s I0720 00:42:45.882194 2700 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:37359 I0720 00:42:45.883267 2700 webserver.cc:374] Webserver started at http://127.0.58.130:37945/ using document root and password file I0720 00:42:45.883956 2700 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb W0720 00:42:45.885702 2835 tablet.cc:1799] T 7e3e1eff26974b46b75eac1cc0290d3b P bc957945380a4e119d7ac829148eede6: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:42:45.888293 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 2700 I0720 00:42:45.894906 2834 heartbeater.cc:324] Connected to a master server at 127.0.58.190:43971 I0720 00:42:45.895103 2834 heartbeater.cc:405] Registering TS with master... I0720 00:42:45.895500 2834 heartbeater.cc:454] Master 127.0.58.190:43971 requested a full tablet report, sending... I0720 00:42:45.897447 2162 ts_manager.cc:161] Re-registered known tserver with Master: bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) W0720 00:42:45.897966 2162 catalog_manager.cc:3988] Tablet ade4b6ce17504569a6d13d9018288194 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) has failed on TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): IO error: some tablet data is in a failed directory W0720 00:42:45.898144 2162 catalog_manager.cc:3988] Tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) has failed on TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): IO error: some tablet data is in a failed directory W0720 00:42:45.898231 2162 catalog_manager.cc:3988] Tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) has failed on TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): IO error: some tablet data is in a failed directory I0720 00:42:45.899286 2162 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:34451 Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 65741a0735184fc583c84c30265e2e4e | 127.0.58.190:43971 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 526eadb7abb3438790a38e0d9973a6a5 | 127.0.58.131:33941 | HEALTHY | bc957945380a4e119d7ac829148eede6 | 127.0.58.130:37359 | HEALTHY | d9d5e137df31450caa9dc831972c1f5e | 127.0.58.129:37035 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.130:37359 env_inject_eio | 1 | hidden | 127.0.58.130:37359 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** | hidden | 127.0.58.130:37359 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:37035 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:37359 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:33941 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:37035 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:37359 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:33941 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING All reported replicas are: A = 526eadb7abb3438790a38e0d9973a6a5 B = bc957945380a4e119d7ac829148eede6 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Tablet 7e3e1eff26974b46b75eac1cc0290d3b of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): RUNNING 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C* | 2 | -1 | Yes C | A B C* | 2 | -1 | Yes Tablet ade4b6ce17504569a6d13d9018288194 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet fc71eee696004a62810a0f8ac4b1185d of table 'test-workload' is conflicted: 2 replicas' active configs disagree with the leader master's 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = 526eadb7abb3438790a38e0d9973a6a5 B = bc957945380a4e119d7ac829148eede6 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 0 | 0 | 3 | 2 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:45.962915 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:46.082520 2834 heartbeater.cc:446] Master 127.0.58.190:43971 was elected leader, sending a full tablet report... W0720 00:42:46.089943 2162 catalog_manager.cc:3988] Tablet ade4b6ce17504569a6d13d9018288194 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) has failed on TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): IO error: some tablet data is in a failed directory W0720 00:42:46.090111 2162 catalog_manager.cc:3988] Tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) has failed on TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): IO error: some tablet data is in a failed directory W0720 00:42:46.090201 2162 catalog_manager.cc:3988] Tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) has failed on TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): IO error: some tablet data is in a failed directory Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 65741a0735184fc583c84c30265e2e4e | 127.0.58.190:43971 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 526eadb7abb3438790a38e0d9973a6a5 | 127.0.58.131:33941 | HEALTHY | bc957945380a4e119d7ac829148eede6 | 127.0.58.130:37359 | HEALTHY | d9d5e137df31450caa9dc831972c1f5e | 127.0.58.129:37035 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.130:37359 env_inject_eio | 1 | hidden | 127.0.58.130:37359 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** | hidden | 127.0.58.130:37359 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:37035 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:37359 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:33941 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:37035 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:37359 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:33941 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING All reported replicas are: A = 526eadb7abb3438790a38e0d9973a6a5 B = bc957945380a4e119d7ac829148eede6 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Tablet ade4b6ce17504569a6d13d9018288194 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 5 | 2 | 0 | 3 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:46.183125 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:46.247426 2687 raft_consensus.cc:922] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Attempting to remove follower bc957945380a4e119d7ac829148eede6 from the Raft config. Reason: The tablet replica hosted on peer bc957945380a4e119d7ac829148eede6 has failed I0720 00:42:46.248138 2687 consensus_queue.cc:206] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:46.249871 2538 raft_consensus.cc:1184] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:46.250447 2888 consensus_queue.cc:984] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:46.252761 2687 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:46.253358 2538 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:46.401434 2888 raft_consensus.cc:922] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Attempting to remove follower bc957945380a4e119d7ac829148eede6 from the Raft config. Reason: The tablet replica hosted on peer bc957945380a4e119d7ac829148eede6 has failed I0720 00:42:46.401968 2888 consensus_queue.cc:206] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:46.403839 2537 raft_consensus.cc:1184] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:42:46.404338 2891 consensus_queue.cc:984] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:42:46.406607 2888 raft_consensus.cc:2792] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:46.407150 2537 raft_consensus.cc:2792] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:46.719218 2891 raft_consensus.cc:472] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc957945380a4e119d7ac829148eede6) I0720 00:42:46.719427 2891 raft_consensus.cc:494] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:46.720013 2891 leader_election.cc:296] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941), bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) I0720 00:42:46.720670 2536 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "526eadb7abb3438790a38e0d9973a6a5" is_pre_election: true I0720 00:42:46.720882 2536 raft_consensus.cc:2311] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9d5e137df31450caa9dc831972c1f5e for term 2 because replica is either leader or believes a valid leader to be alive. I0720 00:42:46.720775 2789 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "d9d5e137df31450caa9dc831972c1f5e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true W0720 00:42:46.721410 2209 leader_election.cc:348] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Illegal state: cannot vote while shut down I0720 00:42:46.721485 2209 leader_election.cc:310] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d9d5e137df31450caa9dc831972c1f5e; no voters: 526eadb7abb3438790a38e0d9973a6a5, bc957945380a4e119d7ac829148eede6 I0720 00:42:46.721706 2891 raft_consensus.cc:2592] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0720 00:42:46.744396 2162 catalog_manager.cc:4067] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e reported cstate change: config changed from index -1 to 2, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:46.752557 2162 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ade4b6ce17504569a6d13d9018288194 on bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) (TS bc957945380a4e119d7ac829148eede6 not found in new config with opid_index 2) I0720 00:42:46.752910 2162 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet ade4b6ce17504569a6d13d9018288194 (attempt 1) I0720 00:42:46.754379 2162 catalog_manager.cc:4067] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 reported cstate change: config changed from index -1 to 2, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:46.754895 2277 consensus_queue.cc:206] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 65741a0735184fc583c84c30265e2e4e | 127.0.58.190:43971 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 526eadb7abb3438790a38e0d9973a6a5 | 127.0.58.131:33941 | HEALTHY | bc957945380a4e119d7ac829148eede6 | 127.0.58.130:37359 | HEALTHY | d9d5e137df31450caa9dc831972c1f5e | 127.0.58.129:37035 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.130:37359 env_inject_eio | 1 | hidden | 127.0.58.130:37359 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** | hidden | 127.0.58.130:37359 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:37035 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:37359 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:33941 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked I0720 00:42:46.757381 2162 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 on bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) (TS bc957945380a4e119d7ac829148eede6 not found in new config with opid_index 2) rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:37035 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:37359 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:33941 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running I0720 00:42:46.757556 2162 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (attempt 1) State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | B C* | 1 | 2 | Yes C | B C* | 1 | 2 | Yes Tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running [LEADER] I0720 00:42:46.757911 2539 raft_consensus.cc:1184] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING All reported replicas are: A = 526eadb7abb3438790a38e0d9973a6a5 B = bc957945380a4e119d7ac829148eede6 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes I0720 00:42:46.758515 2891 consensus_queue.cc:984] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Tablet ade4b6ce17504569a6d13d9018288194 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = bc957945380a4e119d7ac829148eede6 B = 526eadb7abb3438790a38e0d9973a6a5 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | B C* | 1 | 2 | Yes C | B C* | 1 | 2 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 5 | 2 | 0 | 3 | 0 W0720 00:42:46.759207 2209 consensus_peers.cc:458] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 5 First Quartile | 5 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:46.760248 2277 consensus_queue.cc:206] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } I0720 00:42:46.763029 2538 raft_consensus.cc:1184] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) W0720 00:42:46.763186 2209 consensus_peers.cc:458] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:46.763484 2687 consensus_queue.cc:984] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:42:46.763257 2891 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } I0720 00:42:46.765982 2888 raft_consensus.cc:2792] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } I0720 00:42:46.766383 2538 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } I0720 00:42:46.768688 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:46.769953 2768 tablet_service.cc:1066] Processing DeleteTablet for tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 with delete_type TABLET_DATA_TOMBSTONED (TS bc957945380a4e119d7ac829148eede6 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:43322 I0720 00:42:46.769961 2769 tablet_service.cc:1066] Processing DeleteTablet for tablet ade4b6ce17504569a6d13d9018288194 with delete_type TABLET_DATA_TOMBSTONED (TS bc957945380a4e119d7ac829148eede6 not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:43322 I0720 00:42:46.771744 2924 ts_tablet_manager.cc:1400] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:46.774382 2925 ts_tablet_manager.cc:1400] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:42:46.777284 2539 raft_consensus.cc:2792] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } I0720 00:42:46.910609 2916 raft_consensus.cc:472] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bc957945380a4e119d7ac829148eede6) I0720 00:42:46.910841 2916 raft_consensus.cc:494] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:46.912076 2916 leader_election.cc:296] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:46.919039 2789 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "526eadb7abb3438790a38e0d9973a6a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" is_pre_election: true W0720 00:42:46.919677 2471 leader_election.cc:348] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Illegal state: cannot vote while shut down I0720 00:42:46.920632 2277 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "526eadb7abb3438790a38e0d9973a6a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" is_pre_election: true I0720 00:42:46.920811 2277 raft_consensus.cc:2350] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526eadb7abb3438790a38e0d9973a6a5 in term 1. I0720 00:42:46.921241 2473 leader_election.cc:310] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: bc957945380a4e119d7ac829148eede6 I0720 00:42:46.921649 2916 raft_consensus.cc:2644] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:42:46.921727 2916 raft_consensus.cc:472] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bc957945380a4e119d7ac829148eede6) I0720 00:42:46.921794 2916 raft_consensus.cc:2890] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Advancing to term 2 W0720 00:42:47.520910 2891 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 0.757s user 0.003s sys 0.000s W0720 00:42:47.520920 2538 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.754s user 0.002s sys 0.000s I0720 00:42:47.521934 2891 raft_consensus.cc:922] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Attempting to remove follower bc957945380a4e119d7ac829148eede6 from the Raft config. Reason: The tablet replica hosted on peer bc957945380a4e119d7ac829148eede6 has failed I0720 00:42:47.522480 2891 consensus_queue.cc:206] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } W0720 00:42:47.524159 2916 consensus_meta.cc:220] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.602s user 0.000s sys 0.000s I0720 00:42:47.524237 2916 raft_consensus.cc:494] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:47.525032 2146 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet ade4b6ce17504569a6d13d9018288194 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:47.525559 2916 leader_election.cc:296] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 election: Requested vote from peers bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359), d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:47.525624 2789 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "526eadb7abb3438790a38e0d9973a6a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bc957945380a4e119d7ac829148eede6" I0720 00:42:47.526142 2162 catalog_manager.cc:4067] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e reported cstate change: config changed from index 2 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added, now has a pending config: VOTER 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131), VOTER d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } W0720 00:42:47.526924 2471 leader_election.cc:348] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Illegal state: cannot vote while shut down I0720 00:42:47.527281 2276 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f8dcedcbd6bb47ce9de0a37cc84ebca5" candidate_uuid: "526eadb7abb3438790a38e0d9973a6a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" I0720 00:42:47.527456 2276 raft_consensus.cc:2890] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 1 FOLLOWER]: Advancing to term 2 I0720 00:42:47.528879 2536 raft_consensus.cc:1184] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:47.529484 2687 consensus_queue.cc:984] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:47.531450 2923 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:47.532007 2536 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:47.541174 2925 ts_tablet_manager.cc:1413] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:47.541317 2925 log.cc:990] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal/wals/ade4b6ce17504569a6d13d9018288194 I0720 00:42:47.542338 2144 catalog_manager.cc:3437] TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): tablet ade4b6ce17504569a6d13d9018288194 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) successfully deleted W0720 00:42:48.250128 2888 consensus_meta.cc:220] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 1.484s user 0.002s sys 0.000s W0720 00:42:48.250162 2539 consensus_meta.cc:220] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 1.472s user 0.003s sys 0.000s W0720 00:42:48.252655 2276 consensus_meta.cc:220] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 0.725s user 0.000s sys 0.000s I0720 00:42:48.252768 2276 raft_consensus.cc:2350] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526eadb7abb3438790a38e0d9973a6a5 in term 2. W0720 00:42:48.254938 2923 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 0.723s user 0.002s sys 0.000s I0720 00:42:48.255561 2924 ts_tablet_manager.cc:1413] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:42:48.255673 2924 log.cc:990] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P bc957945380a4e119d7ac829148eede6: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/wal/wals/18b33e9892f14d0abfa3d7ecb1ee9e80 W0720 00:42:48.256595 2536 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.724s user 0.001s sys 0.000s I0720 00:42:48.256258 2924 rpcz_store.cc:269] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43322 (request call id 2) took 1486ms. Request Metrics: {"tablet-delete.queue_time_us":1242,"thread_start_us":1125,"threads_started":1} I0720 00:42:48.256808 2159 catalog_manager.cc:4067] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e reported cstate change: config changed from index 3 to 4, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted, no longer has a pending config: VOTER 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131), VOTER d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:48.257743 2144 catalog_manager.cc:3437] TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) successfully deleted I0720 00:42:48.257359 2888 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57296 (request call id 7) took 1497ms. Request Metrics: {"raft.queue_time_us":321,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":171} I0720 00:42:48.258426 2146 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:VOTER RPC for tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1) I0720 00:42:48.258584 2473 leader_election.cc:310] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5, d9d5e137df31450caa9dc831972c1f5e; no voters: bc957945380a4e119d7ac829148eede6 I0720 00:42:48.259176 2277 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57354 (request call id 4) took 1086ms. Request Metrics: {"spinlock_wait_cycles":2497006976} I0720 00:42:48.259860 2687 raft_consensus.cc:922] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Attempting to remove follower bc957945380a4e119d7ac829148eede6 from the Raft config. Reason: The tablet replica hosted on peer bc957945380a4e119d7ac829148eede6 has failed I0720 00:42:48.260363 2687 consensus_queue.cc:206] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:48.262212 2961 raft_consensus.cc:2644] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:42:48.262389 2961 raft_consensus.cc:667] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 2 LEADER]: Becoming Leader. State: Replica: 526eadb7abb3438790a38e0d9973a6a5, State: Running, Role: LEADER I0720 00:42:48.262835 2961 consensus_queue.cc:206] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:48.268383 2159 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ade4b6ce17504569a6d13d9018288194 on bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359) (TS bc957945380a4e119d7ac829148eede6 not found in new config with opid_index 4) I0720 00:42:48.268606 2159 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:VOTER on tablet ade4b6ce17504569a6d13d9018288194 (attempt 1) I0720 00:42:48.269109 2769 tablet_service.cc:1066] Processing DeleteTablet for tablet ade4b6ce17504569a6d13d9018288194 with delete_type TABLET_DATA_TOMBSTONED (TS bc957945380a4e119d7ac829148eede6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43322 I0720 00:42:48.269690 2144 catalog_manager.cc:3437] TS bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): tablet ade4b6ce17504569a6d13d9018288194 (table test-workload [id=4c1b1a3e42344401926664c866bb23b0]) successfully deleted I0720 00:42:48.270761 2277 consensus_queue.cc:206] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } I0720 00:42:48.272778 2536 raft_consensus.cc:1184] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0720 00:42:48.273355 2159 catalog_manager.cc:4067] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e reported cstate change: config changed from index 2 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added, now has a pending config: VOTER 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131), VOTER d9d5e137df31450caa9dc831972c1f5e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:48.273427 2537 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39474 (request call id 4) took 1101ms. Request Metrics: {"spinlock_wait_cycles":2526827136} I0720 00:42:48.274253 2538 raft_consensus.cc:1184] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Refusing update from remote peer d9d5e137df31450caa9dc831972c1f5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:42:48.274387 2687 consensus_queue.cc:984] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0720 00:42:48.274751 2539 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.58.129:48901 (request call id 53) took 1497ms. Request Metrics: {"raft.queue_time_us":375,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"thread_start_us":22168,"threads_started":2,"wal-append.queue_time_us":18829} W0720 00:42:48.274996 2209 consensus_peers.cc:458] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:48.275383 2687 consensus_queue.cc:984] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:42:48.276434 2162 catalog_manager.cc:4067] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 reported cstate change: term changed from 1 to 2, leader changed from bc957945380a4e119d7ac829148eede6 (127.0.58.130) to 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131). New cstate: current_term: 2 leader_uuid: "526eadb7abb3438790a38e0d9973a6a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:48.276998 2888 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } I0720 00:42:48.278662 2537 raft_consensus.cc:2792] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 65741a0735184fc583c84c30265e2e4e | 127.0.58.190:43971 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 526eadb7abb3438790a38e0d9973a6a5 | 127.0.58.131:33941 | HEALTHY | bc957945380a4e119d7ac829148eede6 | 127.0.58.130:37359 | HEALTHY | d9d5e137df31450caa9dc831972c1f5e | 127.0.58.129:37035 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.130:37359 env_inject_eio | 1 | hidden | 127.0.58.130:37359 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** | hidden | 127.0.58.130:37359 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:37035 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:37359 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:33941 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:37035 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:37359 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:33941 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked Tablet Summary Tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING [LEADER] All reported replicas are: A = 526eadb7abb3438790a38e0d9973a6a5 B = d9d5e137df31450caa9dc831972c1f5e C = bc957945380a4e119d7ac829148eede6 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* | | | Yes A | A B* C | 1 | 3 | Yes B | A B* C | 1 | 3 | Yes I0720 00:42:48.281666 2923 raft_consensus.cc:2792] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } Tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 526eadb7abb3438790a38e0d9973a6a5 (127.0.58.131:33941): RUNNING bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035): RUNNING All reported replicas are: A = 526eadb7abb3438790a38e0d9973a6a5 B = bc957945380a4e119d7ac829148eede6 C = d9d5e137df31450caa9dc831972c1f5e The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 2 | -1 | Yes B | [config not available] | | | C | A B C | 2 | -1 | Yes I0720 00:42:48.282181 2538 raft_consensus.cc:2792] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 1 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 3 Median | 5 Third Quartile | 5 Maximum | 5 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 5 Replicas | 13 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:42:48.285115 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:42:48.667629 2277 raft_consensus.cc:1184] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Refusing update from remote peer 526eadb7abb3438790a38e0d9973a6a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0720 00:42:48.668309 2961 consensus_queue.cc:984] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s W0720 00:42:48.673665 2471 consensus_peers.cc:458] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 -> Peer bc957945380a4e119d7ac829148eede6 (127.0.58.130:37359): Couldn't send request to peer bc957945380a4e119d7ac829148eede6. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:42:48.674023 2961 raft_consensus.cc:922] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Attempting to remove follower bc957945380a4e119d7ac829148eede6 from the Raft config. Reason: The tablet replica hosted on peer bc957945380a4e119d7ac829148eede6 has failed I0720 00:42:48.674721 2961 consensus_queue.cc:206] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:48.675925 2277 raft_consensus.cc:1184] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Refusing update from remote peer 526eadb7abb3438790a38e0d9973a6a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0720 00:42:48.676373 2965 consensus_queue.cc:984] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:42:48.678308 2961 raft_consensus.cc:2792] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:48.678822 2277 raft_consensus.cc:2792] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER bc957945380a4e119d7ac829148eede6 (127.0.58.130) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } I0720 00:42:48.787096 2969 ts_tablet_manager.cc:682] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Initiating tablet copy from peer d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:48.787333 2969 tablet_copy_client.cc:204] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:42:48.787505 2969 tablet_copy_client.cc:241] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.129:37035 I0720 00:42:48.793766 2297 tablet_copy_service.cc:135] P d9d5e137df31450caa9dc831972c1f5e: Received BeginTabletCopySession request for tablet ade4b6ce17504569a6d13d9018288194 from peer bc957945380a4e119d7ac829148eede6 ({username='slave'} at 127.0.58.130:35599) I0720 00:42:48.793942 2297 tablet_copy_service.cc:156] P d9d5e137df31450caa9dc831972c1f5e: Beginning new tablet copy session on tablet ade4b6ce17504569a6d13d9018288194 from peer bc957945380a4e119d7ac829148eede6 at {username='slave'} at 127.0.58.130:35599: session id = bc957945380a4e119d7ac829148eede6-ade4b6ce17504569a6d13d9018288194 F0720 00:42:48.836412 2162 quorum_util.cc:167] Check failed: RaftPeerPB::NON_PARTICIPANT != GetConsensusRole(peer_uuid, cstate) (3 vs. 3) Peer bc957945380a4e119d7ac829148eede6 << not a participant in current_term: 1 leader_uuid: "d9d5e137df31450caa9dc831972c1f5e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } } *** Check failure stack trace: *** @ 0x7f527aaea62d google::LogMessage::Fail() at ??:0 @ 0x7f527aaec64c google::LogMessage::SendToLog() at ??:0 @ 0x7f527aaea189 google::LogMessage::Flush() at ??:0 @ 0x7f527aaecfdf google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x7f52828cbc3c kudu::consensus::GetParticipantRole() at ??:0 @ 0x7f528d3f4e93 kudu::master::CatalogManager::BuildLocationsForTablet() at ??:0 @ 0x7f528d3f9e27 kudu::master::CatalogManager::GetTableLocations() at ??:0 @ 0x7f528d543fe6 kudu::master::MasterServiceImpl::GetTableLocations() at ??:0 @ 0x7f5288b49caa std::_Function_handler<>::_M_invoke() at ??:0 @ 0x7f527f34431c std::function<>::operator()() at ??:0 @ 0x7f527f342e8b kudu::rpc::GeneratedServiceIf::Handle() at ??:0 @ 0x7f527f346988 kudu::rpc::ServicePool::RunThread() at ??:0 @ 0x7f527f34beb3 boost::_bi::bind_t<>::operator()() at ??:0 @ 0x7f527f2a680c boost::function0<>::operator()() at ??:0 @ 0x7f527bcf1e0b kudu::Thread::SuperviseThread() at ??:0 @ 0x7f52841a7184 start_thread at ??:0 @ 0x7f5277703ffd clone at ??:0 W0720 00:42:49.039686 2985 meta_cache.cc:770] Network error: LookupRpc { table: 'test-workload', partition-key: (), attempt: 1 } failed: recv got EOF from 127.0.58.190:43971 (error 108) W0720 00:42:49.069905 2537 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.791s user 0.002s sys 0.000s W0720 00:42:49.069927 2888 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 0.792s user 0.000s sys 0.002s W0720 00:42:49.070745 2209 connection.cc:562] Connection torn down before Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:57296 (request call id 8) could send its response I0720 00:42:49.071806 2297 tablet_copy_source_session.cc:202] T ade4b6ce17504569a6d13d9018288194 P d9d5e137df31450caa9dc831972c1f5e: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:42:49.074293 2969 ts_tablet_manager.cc:1400] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Deleting tablet data with delete state TABLET_DATA_COPYING W0720 00:42:49.079519 2582 heartbeater.cc:570] Failed to heartbeat to 127.0.58.190:43971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.58.190:43971: connect: Connection refused (error 111) W0720 00:42:49.099632 2834 heartbeater.cc:570] Failed to heartbeat to 127.0.58.190:43971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.58.190:43971: connect: Connection refused (error 111) W0720 00:42:49.453352 2277 consensus_meta.cc:220] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 0.774s user 0.003s sys 0.000s W0720 00:42:49.453363 2538 consensus_meta.cc:220] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 1.171s user 0.002s sys 0.000s W0720 00:42:49.454290 2961 consensus_meta.cc:220] T f8dcedcbd6bb47ce9de0a37cc84ebca5 P 526eadb7abb3438790a38e0d9973a6a5: Time spent flushing consensus metadata: real 0.776s user 0.003s sys 0.000s W0720 00:42:49.454825 2923 consensus_meta.cc:220] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e: Time spent flushing consensus metadata: real 1.173s user 0.003s sys 0.000s I0720 00:42:49.458860 2538 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.58.129:48901 (request call id 68) took 1176ms. Request Metrics: {"raft.queue_time_us":40,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":4287,"threads_started":1,"wal-append.queue_time_us":4460} Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 65741a0735184fc583c84c30265e2e4e | 127.0.58.190:43971 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 526eadb7abb3438790a38e0d9973a6a5 | 127.0.58.131:33941 | HEALTHY | bc957945380a4e119d7ac829148eede6 | 127.0.58.130:37359 | HEALTHY | d9d5e137df31450caa9dc831972c1f5e | 127.0.58.129:37035 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 3 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.130:37359 env_inject_eio | 1 | hidden | 127.0.58.130:37359 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-1/** | hidden | 127.0.58.130:37359 ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:37035 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:37359 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:33941 never_fsync | true | unsafe,advanced | all 3 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:37035 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:37359 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:33941 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 4 server(s) checked The cluster doesn't have any matching tablets The cluster doesn't have any matching tables Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 0 Median | 0 Third Quartile | 0 Maximum | 0 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:42:49.484397 234 log_verifier.cc:121] Checking tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 I0720 00:42:49.484510 2969 ts_tablet_manager.cc:1413] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:42:49.500281 234 log_verifier.cc:172] Verified matching terms for 4 ops in tablet 18b33e9892f14d0abfa3d7ecb1ee9e80 I0720 00:42:49.500345 234 log_verifier.cc:121] Checking tablet 7e3e1eff26974b46b75eac1cc0290d3b I0720 00:42:49.518692 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 7e3e1eff26974b46b75eac1cc0290d3b I0720 00:42:49.518755 234 log_verifier.cc:121] Checking tablet ade4b6ce17504569a6d13d9018288194 I0720 00:42:49.532200 234 log_verifier.cc:172] Verified matching terms for 5 ops in tablet ade4b6ce17504569a6d13d9018288194 I0720 00:42:49.532260 234 log_verifier.cc:121] Checking tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 I0720 00:42:49.550940 234 log_verifier.cc:172] Verified matching terms for 3 ops in tablet f8dcedcbd6bb47ce9de0a37cc84ebca5 I0720 00:42:49.551002 234 log_verifier.cc:121] Checking tablet fc71eee696004a62810a0f8ac4b1185d I0720 00:42:49.568912 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet fc71eee696004a62810a0f8ac4b1185d I0720 00:42:49.783219 2961 raft_consensus.cc:472] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d9d5e137df31450caa9dc831972c1f5e) I0720 00:42:49.783376 2961 raft_consensus.cc:494] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } I0720 00:42:49.783875 2961 leader_election.cc:296] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d9d5e137df31450caa9dc831972c1f5e (127.0.58.129:37035) I0720 00:42:49.784513 2276 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "18b33e9892f14d0abfa3d7ecb1ee9e80" candidate_uuid: "526eadb7abb3438790a38e0d9973a6a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d9d5e137df31450caa9dc831972c1f5e" is_pre_election: true I0720 00:42:49.784718 2276 raft_consensus.cc:2311] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P d9d5e137df31450caa9dc831972c1f5e [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 526eadb7abb3438790a38e0d9973a6a5 for term 2 because replica is either leader or believes a valid leader to be alive. I0720 00:42:49.785174 2473 leader_election.cc:310] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 526eadb7abb3438790a38e0d9973a6a5; no voters: d9d5e137df31450caa9dc831972c1f5e I0720 00:42:49.785413 2961 raft_consensus.cc:2592] T 18b33e9892f14d0abfa3d7ecb1ee9e80 P 526eadb7abb3438790a38e0d9973a6a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:42:50.383736 2969 consensus_meta.cc:220] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent flushing consensus metadata: real 0.898s user 0.004s sys 0.000s I0720 00:42:50.384328 2969 rpcz_store.cc:269] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.58.129:53251 (request call id 23) took 1597ms. Request Metrics: {"dns_us":56,"tablet-copy.queue_time_us":315,"thread_start_us":190,"threads_started":1} I0720 00:42:50.384708 2969 tablet_copy_client.cc:579] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet copy: Starting download of 0 data blocks... I0720 00:42:50.384996 2969 tablet_copy_client.cc:542] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet copy: Starting download of 1 WAL segments... I0720 00:42:50.386981 2969 tablet_copy_client.cc:418] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:42:51.179970 2969 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Bootstrap starting. I0720 00:42:51.187202 2969 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:42:51.187747 2969 tablet_bootstrap.cc:439] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Bootstrap complete. I0720 00:42:51.187938 2969 ts_tablet_manager.cc:1059] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.005s I0720 00:42:51.188572 2969 raft_consensus.cc:340] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } I0720 00:42:51.188727 2969 raft_consensus.cc:705] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc957945380a4e119d7ac829148eede6, State: Initialized, Role: FOLLOWER I0720 00:42:51.188925 2969 consensus_queue.cc:229] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "526eadb7abb3438790a38e0d9973a6a5" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 33941 } } peers { permanent_uuid: "d9d5e137df31450caa9dc831972c1f5e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 37035 } } peers { permanent_uuid: "bc957945380a4e119d7ac829148eede6" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 37359 } } I0720 00:42:51.189544 2969 ts_tablet_manager.cc:1087] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0720 00:42:51.190430 2297 tablet_copy_service.cc:325] P d9d5e137df31450caa9dc831972c1f5e: Request end of tablet copy session bc957945380a4e119d7ac829148eede6-ade4b6ce17504569a6d13d9018288194 received from {username='slave'} at 127.0.58.130:35599 I0720 00:42:51.190531 2297 tablet_copy_service.cc:413] P d9d5e137df31450caa9dc831972c1f5e: ending tablet copy session bc957945380a4e119d7ac829148eede6-ade4b6ce17504569a6d13d9018288194 on tablet ade4b6ce17504569a6d13d9018288194 with peer bc957945380a4e119d7ac829148eede6 I0720 00:42:51.316581 2789 raft_consensus.cc:1126] T ade4b6ce17504569a6d13d9018288194 P bc957945380a4e119d7ac829148eede6 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[] W0720 00:43:40.531803 2320 heartbeater.cc:570] Failed to heartbeat to 127.0.58.190:43971 (53 consecutive failures): Network error: Failed to ping master at 127.0.58.190:43971: Client connection negotiation failed: client connection to 127.0.58.190:43971: connect: Connection refused (error 111) W0720 00:43:49.401312 2834 heartbeater.cc:570] Failed to heartbeat to 127.0.58.190:43971 (63 consecutive failures): Network error: Failed to ping master at 127.0.58.190:43971: Client connection negotiation failed: client connection to 127.0.58.190:43971: connect: Connection refused (error 111) W0720 00:43:49.549444 2582 heartbeater.cc:570] Failed to heartbeat to 127.0.58.190:43971 (62 consecutive failures): Network error: Failed to ping master at 127.0.58.190:43971: Client connection negotiation failed: client connection to 127.0.58.190:43971: connect: Connection refused (error 111) /data0/jenkins/workspace/kudu-pre-commit-unittest-ASAN/src/kudu/integration-tests/cluster_verifier.cc:118: Failure Failed Bad status: Timed out: Unable to connect to cluster: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.58.190:43971, num_attempts: 344) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.0.58.190:43971: connect: Connection refused (error 111) /data0/jenkins/workspace/kudu-pre-commit-unittest-ASAN/src/kudu/integration-tests/disk_failure-itest.cc:143: Failure Expected: v.CheckRowCount(write_workload.table_name(), ClusterVerifier::AT_LEAST, write_workload.batches_completed()) doesn't generate new fatal failures in the current thread. Actual: it does. I0720 00:43:49.785172 234 external_mini_cluster-itest-base.cc:80] Found fatal failure I0720 00:43:49.785481 234 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 0 with UUID d9d5e137df31450caa9dc831972c1f5e and pid 2196 ************************ BEGIN STACKS ************************** 0x00007fc1df895b9d in ?? () Id Target Id Frame * 1 process 2196 "kudu-tserver" 0x00007fc1df895b9d in ?? () Thread 1 (process 2196): #0 0x00007fc1df895b9d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0720 00:43:50.032261 234 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 1 with UUID bc957945380a4e119d7ac829148eede6 and pid 2700 ************************ BEGIN STACKS ************************** 0x00007fd3a7e6cb9d in ?? () Id Target Id Frame * 1 process 2700 "kudu-tserver" 0x00007fd3a7e6cb9d in ?? () Thread 1 (process 2700): #0 0x00007fd3a7e6cb9d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0720 00:43:50.274740 234 external_mini_cluster-itest-base.cc:86] Attempting to dump stacks of TS 2 with UUID 526eadb7abb3438790a38e0d9973a6a5 and pid 2459 ************************ BEGIN STACKS ************************** 0x00007f4c0c0d0b9d in ?? () Id Target Id Frame * 1 process 2459 "kudu-tserver" 0x00007f4c0c0d0b9d in ?? () Thread 1 (process 2459): #0 0x00007f4c0c0d0b9d in ?? () #1 0x0000000000000000 in ?? () ************************* END STACKS *************************** I0720 00:43:50.515115 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 2196 I0720 00:43:50.520992 2317 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:50.594743 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 2196 I0720 00:43:50.607771 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 2700 W0720 00:43:50.607774 2715 connection.cc:655] server connection from 127.0.58.129:53251 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:43:50.612869 2831 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:50.678761 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 2700 I0720 00:43:50.692114 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 2459 I0720 00:43:50.697511 2579 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:50.767957 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 2459 I0720 00:43:50.783977 234 test_util.cc:135] ----------------------------------------------- I0720 00:43:50.784037 234 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_2.1563583347463414-234 [ FAILED ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2, where GetParam() = ("log", false) (71118 ms) [ RUN ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/3 I0720 00:43:50.786856 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-4 --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:34297 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:50.975860 3312 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:50.976078 3312 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:50.978549 3312 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:43:50.978610 3312 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:43:50.978652 3312 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:50.978693 3312 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:43:50.978752 3312 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:43:50.983192 3312 master_main.cc:98] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:34297 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:50.989624 3312 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 76516us I0720 00:43:50.991050 3312 fs_manager.cc:263] Metadata directory not provided I0720 00:43:50.991118 3312 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:50.991453 3312 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:50.991523 3312 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:50.994168 3312 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:50.994714 3312 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:50.995066 3312 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:50.995405 3312 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-3/instance: uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:50.995750 3312 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-4/instance: uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:50.996104 3312 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.006960 3312 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s I0720 00:43:51.008410 3312 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:51.008939 3312 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:51.012238 3312 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:51.012318 3312 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/wal uuid: "456a7f6ee6f54d268c1570b10309165e" format_stamp: "Formatted at 2019-07-20 00:43:50 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.012447 3312 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:51.027781 3312 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:51.040763 3312 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:34297 I0720 00:43:51.041903 3312 webserver.cc:374] Webserver started at http://127.0.58.190:38971/ using document root and password file I0720 00:43:51.042574 3312 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:43:51.051252 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 3312 I0720 00:43:51.051867 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:43:51.056252 3375 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e: Bootstrap starting. I0720 00:43:51.059975 3375 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e: No blocks or log segments found. Creating new log. I0720 00:43:51.060853 3375 log.cc:525] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e: Log is configured to *not* fsync() on all Append() calls I0720 00:43:51.071481 3375 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e: No bootstrap required, opened a new log I0720 00:43:51.080536 3375 raft_consensus.cc:340] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } I0720 00:43:51.080853 3375 raft_consensus.cc:366] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:51.080937 3375 raft_consensus.cc:705] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 456a7f6ee6f54d268c1570b10309165e, State: Initialized, Role: FOLLOWER I0720 00:43:51.081514 3375 consensus_queue.cc:229] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } I0720 00:43:51.081748 3375 raft_consensus.cc:380] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:43:51.081825 3375 raft_consensus.cc:472] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:43:51.081952 3375 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:51.084328 3375 raft_consensus.cc:494] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } I0720 00:43:51.084868 3375 leader_election.cc:310] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 456a7f6ee6f54d268c1570b10309165e; no voters: I0720 00:43:51.086324 3375 leader_election.cc:296] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:43:51.086496 3378 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:51.087044 3378 raft_consensus.cc:667] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [term 1 LEADER]: Becoming Leader. State: Replica: 456a7f6ee6f54d268c1570b10309165e, State: Running, Role: LEADER I0720 00:43:51.087466 3378 consensus_queue.cc:206] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } I0720 00:43:51.088982 3378 sys_catalog.cc:340] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 456a7f6ee6f54d268c1570b10309165e. Latest consensus state: current_term: 1 leader_uuid: "456a7f6ee6f54d268c1570b10309165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } } I0720 00:43:51.089294 3378 sys_catalog.cc:343] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: This master's current role is: LEADER I0720 00:43:51.089645 3380 sys_catalog.cc:340] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "456a7f6ee6f54d268c1570b10309165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } } I0720 00:43:51.090502 3380 sys_catalog.cc:343] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: This master's current role is: LEADER I0720 00:43:51.090704 3375 sys_catalog.cc:340] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "456a7f6ee6f54d268c1570b10309165e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "456a7f6ee6f54d268c1570b10309165e" member_type: VOTER } } I0720 00:43:51.090857 3375 sys_catalog.cc:343] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: This master's current role is: LEADER I0720 00:43:51.092644 3375 sys_catalog.cc:428] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e [sys.catalog]: configured and running, proceeding with master startup. I0720 00:43:51.092986 3382 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:43:51.094365 3384 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0720 00:43:51.100875 3382 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:43:51.179836 3382 catalog_manager.cc:959] Generated new certificate authority record I0720 00:43:51.181452 3382 catalog_manager.cc:1084] Loading token signing keys... I0720 00:43:51.191356 3382 catalog_manager.cc:4371] T 00000000000000000000000000000000 P 456a7f6ee6f54d268c1570b10309165e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:51.232169 3376 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:51.232378 3376 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:51.234488 3376 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:43:51.234539 3376 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:51.234663 3376 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:43:51.250409 3376 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:51.258234 3376 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 77016us I0720 00:43:51.258981 3376 fs_manager.cc:263] Metadata directory not provided I0720 00:43:51.259039 3376 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:51.259368 3376 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:51.259438 3376 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:51.261693 3376 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.262262 3376 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.262614 3376 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.262921 3376 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3/instance: uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.263267 3376 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4/instance: uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.263581 3376 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.286094 3376 fs_manager.cc:503] Time spent creating directory manager: real 0.022s user 0.009s sys 0.012s I0720 00:43:51.287636 3376 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:51.288166 3376 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:51.291436 3376 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:51.291509 3376 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.291668 3376 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:51.358736 3376 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:51.363044 3376 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:51.363126 3376 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:51.400542 3376 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:39177 I0720 00:43:51.401585 3376 webserver.cc:374] Webserver started at http://127.0.58.129:42651/ using document root and password file I0720 00:43:51.402298 3376 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:43:51.412434 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 3376 I0720 00:43:51.413053 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:43:51.418015 3509 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34297 I0720 00:43:51.419170 3509 heartbeater.cc:405] Registering TS with master... I0720 00:43:51.419646 3509 heartbeater.cc:454] Master 127.0.58.190:34297 requested a full tablet report, sending... I0720 00:43:51.421293 3342 ts_manager.cc:161] Registered new tserver with Master: 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) I0720 00:43:51.422827 3342 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:56967 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:51.563030 3513 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:51.563254 3513 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:51.565294 3513 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:43:51.565357 3513 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:51.565464 3513 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:43:51.569277 3513 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:51.576647 3513 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 77016us I0720 00:43:51.577366 3513 fs_manager.cc:263] Metadata directory not provided I0720 00:43:51.577430 3513 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:51.577766 3513 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:51.577833 3513 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:51.580046 3513 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.580564 3513 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.580912 3513 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.581223 3513 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-3/instance: uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.581570 3513 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-4/instance: uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.581897 3513 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.592567 3513 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.003s I0720 00:43:51.594013 3513 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:51.594528 3513 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:51.597826 3513 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:51.597903 3513 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/wal uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.598057 3513 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:51.647632 3513 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:51.649212 3513 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:51.649296 3513 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:51.674211 3513 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:46827 I0720 00:43:51.675240 3513 webserver.cc:374] Webserver started at http://127.0.58.130:36639/ using document root and password file I0720 00:43:51.675868 3513 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:43:51.678223 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 3513 I0720 00:43:51.679011 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:43:51.690018 3636 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34297 I0720 00:43:51.690284 3636 heartbeater.cc:405] Registering TS with master... I0720 00:43:51.690850 3636 heartbeater.cc:454] Master 127.0.58.190:34297 requested a full tablet report, sending... I0720 00:43:51.692095 3342 ts_manager.cc:161] Registered new tserver with Master: aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827) I0720 00:43:51.692997 3342 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:53941 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:51.831971 3639 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:51.832206 3639 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:51.834280 3639 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:43:51.834342 3639 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:51.834451 3639 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:43:51.838220 3639 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:51.845577 3639 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 77016us I0720 00:43:51.846374 3639 fs_manager.cc:263] Metadata directory not provided I0720 00:43:51.846438 3639 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:51.846751 3639 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:51.846812 3639 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:51.848968 3639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.849462 3639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.849815 3639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.850169 3639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-3/instance: uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.850508 3639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-4/instance: uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.850836 3639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.860736 3639 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.004s sys 0.005s I0720 00:43:51.862171 3639 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:51.862653 3639 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:51.866050 3639 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:51.866168 3639 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/wal uuid: "68ef7d6143504eeba96ee091e0d753d3" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:51.866305 3639 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:51.914544 3639 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:51.916080 3639 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:51.916162 3639 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:51.939239 3639 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:43161 I0720 00:43:51.940378 3639 webserver.cc:374] Webserver started at http://127.0.58.131:40769/ using document root and password file I0720 00:43:51.942324 3639 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:43:51.946974 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 3639 I0720 00:43:51.947770 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.132:0 --local_ip_for_outbound_sockets=127.0.58.132 --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0720 00:43:51.957357 3762 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34297 I0720 00:43:51.957566 3762 heartbeater.cc:405] Registering TS with master... I0720 00:43:51.958066 3762 heartbeater.cc:454] Master 127.0.58.190:34297 requested a full tablet report, sending... I0720 00:43:51.959283 3342 ts_manager.cc:161] Registered new tserver with Master: 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161) I0720 00:43:51.960361 3342 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:33071 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:52.103916 3765 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:52.104132 3765 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:52.106271 3765 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:43:52.106331 3765 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:52.106432 3765 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.132 I0720 00:43:52.110373 3765 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34297 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.132 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:52.118099 3765 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 77516us I0720 00:43:52.118880 3765 fs_manager.cc:263] Metadata directory not provided I0720 00:43:52.118940 3765 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:52.119271 3765 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:52.119334 3765 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:52.121511 3765 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/instance: uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.122021 3765 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-1/instance: uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.122382 3765 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-2/instance: uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.122683 3765 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-3/instance: uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.122987 3765 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-4/instance: uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.123308 3765 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/wal/instance: uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.133778 3765 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.005s sys 0.005s I0720 00:43:52.135347 3765 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:52.135854 3765 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:52.139000 3765 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:52.139075 3765 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/wal uuid: "5af276af9d674086b48b969af0dff3fc" format_stamp: "Formatted at 2019-07-20 00:43:52 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.139217 3765 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/wal 5 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:52.172163 3765 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:52.173820 3765 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:52.173888 3765 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:52.197131 3765 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.132:34605 I0720 00:43:52.198261 3765 webserver.cc:374] Webserver started at http://127.0.58.132:39451/ using document root and password file I0720 00:43:52.198909 3765 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb I0720 00:43:52.205988 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 3765 I0720 00:43:52.212306 3888 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34297 I0720 00:43:52.212522 3888 heartbeater.cc:405] Registering TS with master... I0720 00:43:52.213066 3888 heartbeater.cc:454] Master 127.0.58.190:34297 requested a full tablet report, sending... I0720 00:43:52.214131 3342 ts_manager.cc:161] Registered new tserver with Master: 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) I0720 00:43:52.214959 3342 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.132:52731 I0720 00:43:52.215708 234 external_mini_cluster.cc:628] 4 TS(s) registered with all masters I0720 00:43:52.229959 234 test_util.cc:212] Using random seed: 588121133 I0720 00:43:52.245060 3341 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37030: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\231\231\231\031\004\001\0002333\004\001\000\313\314\314L\004\001\000dfff" } partition_schema { range_schema { columns { name: "key" } } } I0720 00:43:52.299526 3699 tablet_service.cc:1022] Processing CreateTablet for tablet 707b2290dfb44e3caa26d19101fd8b3e (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:43:52.302261 3573 tablet_service.cc:1022] Processing CreateTablet for tablet 983da8448c254fb690777c7dbccbf7fa (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:43:52.302680 3824 tablet_service.cc:1022] Processing CreateTablet for tablet 7548db0376b04d4b8ca358bcddb84fd5 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:43:52.301671 3446 tablet_service.cc:1022] Processing CreateTablet for tablet 983da8448c254fb690777c7dbccbf7fa (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:43:52.303200 3443 tablet_service.cc:1022] Processing CreateTablet for tablet c137ad158a18463694d366278635f9b4 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:43:52.302829 3444 tablet_service.cc:1022] Processing CreateTablet for tablet 7548db0376b04d4b8ca358bcddb84fd5 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:43:52.302433 3445 tablet_service.cc:1022] Processing CreateTablet for tablet 707b2290dfb44e3caa26d19101fd8b3e (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:43:52.306886 3825 tablet_service.cc:1022] Processing CreateTablet for tablet 983da8448c254fb690777c7dbccbf7fa (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION VALUES < 429496729 I0720 00:43:52.307802 3823 tablet_service.cc:1022] Processing CreateTablet for tablet 6042d34aeef84544bb6535ed8ca52ac3 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:43:52.310773 3572 tablet_service.cc:1022] Processing CreateTablet for tablet 707b2290dfb44e3caa26d19101fd8b3e (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 429496729 <= VALUES < 858993458 I0720 00:43:52.310750 3698 tablet_service.cc:1022] Processing CreateTablet for tablet 7548db0376b04d4b8ca358bcddb84fd5 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 858993458 <= VALUES < 1288490187 I0720 00:43:52.317198 3697 tablet_service.cc:1022] Processing CreateTablet for tablet 6042d34aeef84544bb6535ed8ca52ac3 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:43:52.319010 3571 tablet_service.cc:1022] Processing CreateTablet for tablet 6042d34aeef84544bb6535ed8ca52ac3 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION 1288490187 <= VALUES < 1717986916 I0720 00:43:52.325632 3570 tablet_service.cc:1022] Processing CreateTablet for tablet c137ad158a18463694d366278635f9b4 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:43:52.327172 3696 tablet_service.cc:1022] Processing CreateTablet for tablet c137ad158a18463694d366278635f9b4 (table=test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]), partition=RANGE (key) PARTITION VALUES >= 1717986916 I0720 00:43:52.329839 3915 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4: Bootstrap starting. I0720 00:43:52.330444 3916 tablet_bootstrap.cc:439] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4: Bootstrap starting. I0720 00:43:52.332636 3916 tablet_bootstrap.cc:590] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4: No blocks or log segments found. Creating new log. I0720 00:43:52.333623 3918 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:52.335709 3918 tablet_bootstrap.cc:590] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: No blocks or log segments found. Creating new log. I0720 00:43:52.337003 3919 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc: Bootstrap starting. I0720 00:43:52.338963 3919 tablet_bootstrap.cc:590] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc: No blocks or log segments found. Creating new log. I0720 00:43:52.340617 3917 tablet_bootstrap.cc:439] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: Bootstrap starting. I0720 00:43:52.343015 3925 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc: Bootstrap starting. I0720 00:43:52.339663 3921 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4: Bootstrap starting. I0720 00:43:52.347169 3917 tablet_bootstrap.cc:590] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: No blocks or log segments found. Creating new log. I0720 00:43:52.347748 3921 tablet_bootstrap.cc:590] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4: No blocks or log segments found. Creating new log. I0720 00:43:52.347867 3917 log.cc:525] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: Log is configured to *not* fsync() on all Append() calls I0720 00:43:52.348572 3921 log.cc:525] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4: Log is configured to *not* fsync() on all Append() calls I0720 00:43:52.349432 3925 tablet_bootstrap.cc:590] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc: No blocks or log segments found. Creating new log. I0720 00:43:52.344296 3915 tablet_bootstrap.cc:590] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4: No blocks or log segments found. Creating new log. I0720 00:43:52.339345 3920 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:52.339953 3922 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:52.352073 3917 tablet_bootstrap.cc:439] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: No bootstrap required, opened a new log I0720 00:43:52.352955 3922 tablet_bootstrap.cc:590] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: No blocks or log segments found. Creating new log. I0720 00:43:52.353256 3917 ts_tablet_manager.cc:1059] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0720 00:43:52.353924 3919 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc: No bootstrap required, opened a new log I0720 00:43:52.353972 3922 log.cc:525] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Log is configured to *not* fsync() on all Append() calls I0720 00:43:52.354107 3919 ts_tablet_manager.cc:1059] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s I0720 00:43:52.358206 3925 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc: No bootstrap required, opened a new log I0720 00:43:52.359671 3925 ts_tablet_manager.cc:1059] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s I0720 00:43:52.340211 3923 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:52.365456 3932 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3: Bootstrap starting. I0720 00:43:52.366571 3933 tablet_bootstrap.cc:439] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3: Bootstrap starting. I0720 00:43:52.366578 3923 tablet_bootstrap.cc:590] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: No blocks or log segments found. Creating new log. I0720 00:43:52.363441 3920 tablet_bootstrap.cc:590] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: No blocks or log segments found. Creating new log. I0720 00:43:52.368561 3936 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3: Bootstrap starting. I0720 00:43:52.369931 3921 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4: No bootstrap required, opened a new log I0720 00:43:52.370569 3924 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: Bootstrap starting. I0720 00:43:52.371107 3931 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: Bootstrap starting. I0720 00:43:52.372172 3932 tablet_bootstrap.cc:590] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3: No blocks or log segments found. Creating new log. I0720 00:43:52.372182 3924 tablet_bootstrap.cc:590] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: No blocks or log segments found. Creating new log. I0720 00:43:52.371759 3936 tablet_bootstrap.cc:590] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3: No blocks or log segments found. Creating new log. I0720 00:43:52.371723 3933 tablet_bootstrap.cc:590] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3: No blocks or log segments found. Creating new log. I0720 00:43:52.373808 3932 log.cc:525] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3: Log is configured to *not* fsync() on all Append() calls I0720 00:43:52.375432 3915 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4: No bootstrap required, opened a new log I0720 00:43:52.375628 3931 tablet_bootstrap.cc:590] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: No blocks or log segments found. Creating new log. I0720 00:43:52.376005 3915 ts_tablet_manager.cc:1059] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s I0720 00:43:52.376258 3923 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: No bootstrap required, opened a new log I0720 00:43:52.376163 3919 raft_consensus.cc:340] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.377063 3919 raft_consensus.cc:366] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.377458 3919 raft_consensus.cc:705] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af276af9d674086b48b969af0dff3fc, State: Initialized, Role: FOLLOWER I0720 00:43:52.377673 3923 ts_tablet_manager.cc:1059] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s I0720 00:43:52.377249 3917 raft_consensus.cc:340] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.377949 3917 raft_consensus.cc:366] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.378036 3917 raft_consensus.cc:705] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af276af9d674086b48b969af0dff3fc, State: Initialized, Role: FOLLOWER I0720 00:43:52.377138 3920 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: No bootstrap required, opened a new log I0720 00:43:52.378186 3921 ts_tablet_manager.cc:1059] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s I0720 00:43:52.378222 3920 ts_tablet_manager.cc:1059] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s I0720 00:43:52.378566 3917 consensus_queue.cc:229] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.377243 3932 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3: No bootstrap required, opened a new log I0720 00:43:52.379029 3932 ts_tablet_manager.cc:1059] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0720 00:43:52.376821 3925 raft_consensus.cc:340] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.379202 3925 raft_consensus.cc:366] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.379333 3925 raft_consensus.cc:705] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af276af9d674086b48b969af0dff3fc, State: Initialized, Role: FOLLOWER I0720 00:43:52.379487 3916 tablet_bootstrap.cc:439] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4: No bootstrap required, opened a new log I0720 00:43:52.379390 3919 consensus_queue.cc:229] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.379675 3916 ts_tablet_manager.cc:1059] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.003s I0720 00:43:52.380218 3925 consensus_queue.cc:229] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.380425 3888 heartbeater.cc:446] Master 127.0.58.190:34297 was elected leader, sending a full tablet report... I0720 00:43:52.377805 3936 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3: No bootstrap required, opened a new log I0720 00:43:52.376591 3922 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: No bootstrap required, opened a new log I0720 00:43:52.382230 3922 ts_tablet_manager.cc:1059] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s I0720 00:43:52.382249 3936 ts_tablet_manager.cc:1059] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s I0720 00:43:52.384709 3917 ts_tablet_manager.cc:1087] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s I0720 00:43:52.386869 3924 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: No bootstrap required, opened a new log I0720 00:43:52.386982 3924 ts_tablet_manager.cc:1059] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s I0720 00:43:52.387060 3918 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: No bootstrap required, opened a new log I0720 00:43:52.387194 3918 ts_tablet_manager.cc:1059] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.000s I0720 00:43:52.380089 3933 tablet_bootstrap.cc:439] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3: No bootstrap required, opened a new log I0720 00:43:52.387995 3933 ts_tablet_manager.cc:1059] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s I0720 00:43:52.388828 3925 ts_tablet_manager.cc:1087] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc: Time spent starting tablet: real 0.029s user 0.000s sys 0.007s I0720 00:43:52.389645 3919 ts_tablet_manager.cc:1087] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc: Time spent starting tablet: real 0.035s user 0.005s sys 0.005s I0720 00:43:52.391412 3931 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: No bootstrap required, opened a new log I0720 00:43:52.392614 3931 ts_tablet_manager.cc:1059] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s I0720 00:43:52.394101 3918 raft_consensus.cc:340] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.394232 3923 raft_consensus.cc:340] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.394729 3918 raft_consensus.cc:366] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.394469 3922 raft_consensus.cc:340] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.394820 3923 raft_consensus.cc:366] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.395174 3922 raft_consensus.cc:366] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.395217 3923 raft_consensus.cc:705] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: FOLLOWER I0720 00:43:52.395260 3922 raft_consensus.cc:705] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: FOLLOWER I0720 00:43:52.394919 3918 raft_consensus.cc:705] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: FOLLOWER I0720 00:43:52.395893 3923 consensus_queue.cc:229] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.396193 3922 consensus_queue.cc:229] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.394021 3920 raft_consensus.cc:340] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.396502 3920 raft_consensus.cc:366] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.396595 3920 raft_consensus.cc:705] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: FOLLOWER I0720 00:43:52.395897 3918 consensus_queue.cc:229] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.398574 3509 heartbeater.cc:446] Master 127.0.58.190:34297 was elected leader, sending a full tablet report... I0720 00:43:52.398613 3921 raft_consensus.cc:340] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.399132 3921 raft_consensus.cc:366] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.399178 3923 ts_tablet_manager.cc:1087] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s I0720 00:43:52.399507 3921 raft_consensus.cc:705] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa238bcb02fb407ebb4fdca6d3865dd4, State: Initialized, Role: FOLLOWER I0720 00:43:52.398972 3920 consensus_queue.cc:229] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.399271 3916 raft_consensus.cc:340] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.400020 3916 raft_consensus.cc:366] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.400116 3916 raft_consensus.cc:705] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa238bcb02fb407ebb4fdca6d3865dd4, State: Initialized, Role: FOLLOWER I0720 00:43:52.400549 3916 consensus_queue.cc:229] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.401458 3924 raft_consensus.cc:340] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.401732 3924 raft_consensus.cc:366] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.401820 3924 raft_consensus.cc:705] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa238bcb02fb407ebb4fdca6d3865dd4, State: Initialized, Role: FOLLOWER I0720 00:43:52.402050 3924 consensus_queue.cc:229] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.400912 3921 consensus_queue.cc:229] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.403396 3636 heartbeater.cc:446] Master 127.0.58.190:34297 was elected leader, sending a full tablet report... I0720 00:43:52.404004 3920 ts_tablet_manager.cc:1087] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.025s user 0.009s sys 0.000s W0720 00:43:52.404553 3510 tablet.cc:1799] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:43:52.405166 3918 ts_tablet_manager.cc:1087] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s I0720 00:43:52.405385 3922 ts_tablet_manager.cc:1087] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s I0720 00:43:52.407310 3915 raft_consensus.cc:340] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.407649 3921 ts_tablet_manager.cc:1087] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent starting tablet: real 0.029s user 0.002s sys 0.006s I0720 00:43:52.408113 3915 raft_consensus.cc:366] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.408277 3915 raft_consensus.cc:705] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa238bcb02fb407ebb4fdca6d3865dd4, State: Initialized, Role: FOLLOWER I0720 00:43:52.408442 3916 ts_tablet_manager.cc:1087] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent starting tablet: real 0.029s user 0.000s sys 0.008s I0720 00:43:52.408557 3915 consensus_queue.cc:229] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.408529 3924 ts_tablet_manager.cc:1087] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent starting tablet: real 0.021s user 0.001s sys 0.004s I0720 00:43:52.409081 3915 ts_tablet_manager.cc:1087] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4: Time spent starting tablet: real 0.033s user 0.004s sys 0.005s I0720 00:43:52.413007 3932 raft_consensus.cc:340] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.413369 3932 raft_consensus.cc:366] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.413424 3932 raft_consensus.cc:705] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ef7d6143504eeba96ee091e0d753d3, State: Initialized, Role: FOLLOWER I0720 00:43:52.413848 3932 consensus_queue.cc:229] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.414237 3936 raft_consensus.cc:340] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.414384 3936 raft_consensus.cc:366] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.414425 3936 raft_consensus.cc:705] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ef7d6143504eeba96ee091e0d753d3, State: Initialized, Role: FOLLOWER I0720 00:43:52.414568 3936 consensus_queue.cc:229] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.414945 3933 raft_consensus.cc:340] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.415088 3933 raft_consensus.cc:366] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.415127 3933 raft_consensus.cc:705] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ef7d6143504eeba96ee091e0d753d3, State: Initialized, Role: FOLLOWER I0720 00:43:52.415249 3933 consensus_queue.cc:229] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.415570 3931 raft_consensus.cc:340] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.415702 3931 raft_consensus.cc:366] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:52.415740 3931 raft_consensus.cc:705] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ef7d6143504eeba96ee091e0d753d3, State: Initialized, Role: FOLLOWER I0720 00:43:52.415864 3931 consensus_queue.cc:229] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.416538 3762 heartbeater.cc:446] Master 127.0.58.190:34297 was elected leader, sending a full tablet report... I0720 00:43:52.417217 3933 ts_tablet_manager.cc:1087] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3: Time spent starting tablet: real 0.029s user 0.000s sys 0.005s I0720 00:43:52.417390 3931 ts_tablet_manager.cc:1087] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: Time spent starting tablet: real 0.024s user 0.000s sys 0.005s I0720 00:43:52.417762 3932 ts_tablet_manager.cc:1087] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3: Time spent starting tablet: real 0.037s user 0.003s sys 0.005s I0720 00:43:52.418025 3936 ts_tablet_manager.cc:1087] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3: Time spent starting tablet: real 0.036s user 0.005s sys 0.001s W0720 00:43:52.427817 3637 tablet.cc:1799] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:43:52.444607 3763 tablet.cc:1799] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:43:52.444726 3951 raft_consensus.cc:472] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:52.444928 3951 raft_consensus.cc:494] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.446360 3951 leader_election.cc:296] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177), 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) W0720 00:43:52.450639 3889 tablet.cc:1799] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:43:52.453274 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "983da8448c254fb690777c7dbccbf7fa" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" is_pre_election: true I0720 00:43:52.453281 3466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "983da8448c254fb690777c7dbccbf7fa" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c6738598e748b383992ddac5e2c2ca" is_pre_election: true I0720 00:43:52.453595 3845 raft_consensus.cc:2350] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 0. I0720 00:43:52.453595 3466 raft_consensus.cc:2350] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 0. I0720 00:43:52.454160 3525 leader_election.cc:310] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af276af9d674086b48b969af0dff3fc, aa238bcb02fb407ebb4fdca6d3865dd4; no voters: I0720 00:43:52.454562 3951 raft_consensus.cc:2644] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:52.454660 3951 raft_consensus.cc:472] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:52.454717 3951 raft_consensus.cc:2890] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.456363 3951 raft_consensus.cc:494] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.456866 3951 leader_election.cc:296] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177), 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) I0720 00:43:52.457346 3466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "983da8448c254fb690777c7dbccbf7fa" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c6738598e748b383992ddac5e2c2ca" I0720 00:43:52.457348 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "983da8448c254fb690777c7dbccbf7fa" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" I0720 00:43:52.457473 3466 raft_consensus.cc:2890] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.457479 3845 raft_consensus.cc:2890] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.459131 3845 raft_consensus.cc:2350] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 1. I0720 00:43:52.459164 3466 raft_consensus.cc:2350] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 1. I0720 00:43:52.459460 3525 leader_election.cc:310] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af276af9d674086b48b969af0dff3fc, aa238bcb02fb407ebb4fdca6d3865dd4; no voters: I0720 00:43:52.459725 3951 raft_consensus.cc:2644] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:52.459862 3951 raft_consensus.cc:667] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Becoming Leader. State: Replica: aa238bcb02fb407ebb4fdca6d3865dd4, State: Running, Role: LEADER I0720 00:43:52.460220 3953 raft_consensus.cc:472] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:52.460449 3953 raft_consensus.cc:494] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.460427 3951 consensus_queue.cc:206] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:52.461755 3953 leader_election.cc:296] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605), 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161) I0720 00:43:52.461984 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6042d34aeef84544bb6535ed8ca52ac3" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" is_pre_election: true I0720 00:43:52.462180 3845 raft_consensus.cc:2350] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 0. I0720 00:43:52.462620 3525 leader_election.cc:310] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af276af9d674086b48b969af0dff3fc, aa238bcb02fb407ebb4fdca6d3865dd4; no voters: I0720 00:43:52.462962 3951 raft_consensus.cc:2644] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:52.463110 3951 raft_consensus.cc:472] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:52.463214 3951 raft_consensus.cc:2890] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.465342 3951 raft_consensus.cc:494] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.464916 3342 catalog_manager.cc:4067] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 reported cstate change: term changed from 0 to 1, leader changed from to aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } } I0720 00:43:52.466225 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6042d34aeef84544bb6535ed8ca52ac3" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" I0720 00:43:52.466400 3845 raft_consensus.cc:2890] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.467236 3719 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6042d34aeef84544bb6535ed8ca52ac3" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef7d6143504eeba96ee091e0d753d3" is_pre_election: true I0720 00:43:52.467520 3719 raft_consensus.cc:2350] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 0. I0720 00:43:52.468286 3950 raft_consensus.cc:472] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:52.468443 3845 raft_consensus.cc:2350] T 6042d34aeef84544bb6535ed8ca52ac3 P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 1. I0720 00:43:52.468509 3950 raft_consensus.cc:494] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.468780 3525 leader_election.cc:310] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5af276af9d674086b48b969af0dff3fc, aa238bcb02fb407ebb4fdca6d3865dd4; no voters: I0720 00:43:52.469095 3953 raft_consensus.cc:2644] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:52.469143 3719 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6042d34aeef84544bb6535ed8ca52ac3" candidate_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef7d6143504eeba96ee091e0d753d3" I0720 00:43:52.469255 3953 raft_consensus.cc:667] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Becoming Leader. State: Replica: aa238bcb02fb407ebb4fdca6d3865dd4, State: Running, Role: LEADER I0720 00:43:52.469288 3719 raft_consensus.cc:2890] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.469429 3953 consensus_queue.cc:206] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.471626 3719 raft_consensus.cc:2350] T 6042d34aeef84544bb6535ed8ca52ac3 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa238bcb02fb407ebb4fdca6d3865dd4 in term 1. I0720 00:43:52.471854 3951 leader_election.cc:296] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605), 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161) I0720 00:43:52.472982 3341 catalog_manager.cc:4067] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 reported cstate change: term changed from 0 to 1, leader changed from to aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: UNKNOWN } } } I0720 00:43:52.483497 3950 leader_election.cc:296] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161), 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) I0720 00:43:52.494558 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" is_pre_election: true I0720 00:43:52.494570 3719 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef7d6143504eeba96ee091e0d753d3" is_pre_election: true I0720 00:43:52.494760 3845 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 0. I0720 00:43:52.494777 3719 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 0. I0720 00:43:52.495185 3398 leader_election.cc:310] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 5af276af9d674086b48b969af0dff3fc, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:52.495666 3950 raft_consensus.cc:2644] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:52.495741 3950 raft_consensus.cc:472] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:52.495833 3950 raft_consensus.cc:2890] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.497742 3950 raft_consensus.cc:494] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.498293 3950 leader_election.cc:296] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 1 election: Requested vote from peers 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161), 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) I0720 00:43:52.498790 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" I0720 00:43:52.498776 3719 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ef7d6143504eeba96ee091e0d753d3" I0720 00:43:52.498967 3845 raft_consensus.cc:2890] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.498994 3719 raft_consensus.cc:2890] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.501070 3719 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 1. I0720 00:43:52.501083 3845 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 1. I0720 00:43:52.501487 3399 leader_election.cc:310] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:52.501765 3950 raft_consensus.cc:2644] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:52.501910 3950 raft_consensus.cc:667] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 LEADER]: Becoming Leader. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Running, Role: LEADER I0720 00:43:52.502374 3950 consensus_queue.cc:206] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.506064 3341 catalog_manager.cc:4067] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca reported cstate change: term changed from 0 to 1, leader changed from to 16c6738598e748b383992ddac5e2c2ca (127.0.58.129). New cstate: current_term: 1 leader_uuid: "16c6738598e748b383992ddac5e2c2ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } health_report { overall_health: HEALTHY } } } I0720 00:43:52.513783 3955 raft_consensus.cc:472] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:52.513931 3955 raft_consensus.cc:494] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.515136 3955 leader_election.cc:296] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827), 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) I0720 00:43:52.526401 3466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c137ad158a18463694d366278635f9b4" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c6738598e748b383992ddac5e2c2ca" is_pre_election: true I0720 00:43:52.526618 3466 raft_consensus.cc:2350] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 0. I0720 00:43:52.526546 3593 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c137ad158a18463694d366278635f9b4" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" is_pre_election: true I0720 00:43:52.526907 3593 raft_consensus.cc:2350] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 0. I0720 00:43:52.527014 3651 leader_election.cc:310] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:52.527474 3955 raft_consensus.cc:2644] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:52.527552 3955 raft_consensus.cc:472] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:52.527618 3955 raft_consensus.cc:2890] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.529224 3955 raft_consensus.cc:494] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.529717 3955 leader_election.cc:296] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 election: Requested vote from peers aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827), 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) I0720 00:43:52.530220 3593 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c137ad158a18463694d366278635f9b4" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" I0720 00:43:52.530359 3593 raft_consensus.cc:2890] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.530324 3466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c137ad158a18463694d366278635f9b4" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c6738598e748b383992ddac5e2c2ca" I0720 00:43:52.530506 3466 raft_consensus.cc:2890] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.531986 3593 raft_consensus.cc:2350] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 1. I0720 00:43:52.532317 3650 leader_election.cc:310] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ef7d6143504eeba96ee091e0d753d3, aa238bcb02fb407ebb4fdca6d3865dd4; no voters: I0720 00:43:52.532375 3466 raft_consensus.cc:2350] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 1. I0720 00:43:52.532588 3955 raft_consensus.cc:2644] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:52.532712 3955 raft_consensus.cc:667] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 LEADER]: Becoming Leader. State: Replica: 68ef7d6143504eeba96ee091e0d753d3, State: Running, Role: LEADER I0720 00:43:52.533077 3955 consensus_queue.cc:206] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.536712 3341 catalog_manager.cc:4067] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 reported cstate change: term changed from 0 to 1, leader changed from to 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131). New cstate: current_term: 1 leader_uuid: "68ef7d6143504eeba96ee091e0d753d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } health_report { overall_health: UNKNOWN } } } I0720 00:43:52.576421 3955 raft_consensus.cc:472] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:52.576596 3955 raft_consensus.cc:494] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.577203 3955 leader_election.cc:296] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177), aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827) I0720 00:43:52.577791 3593 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "707b2290dfb44e3caa26d19101fd8b3e" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" is_pre_election: true I0720 00:43:52.577798 3466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "707b2290dfb44e3caa26d19101fd8b3e" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c6738598e748b383992ddac5e2c2ca" is_pre_election: true I0720 00:43:52.577996 3466 raft_consensus.cc:2350] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 0. I0720 00:43:52.577986 3593 raft_consensus.cc:2350] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 0. I0720 00:43:52.578388 3651 leader_election.cc:310] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:52.578675 3955 raft_consensus.cc:2644] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:52.578735 3955 raft_consensus.cc:472] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:52.578794 3955 raft_consensus.cc:2890] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.580334 3955 raft_consensus.cc:494] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.580857 3955 leader_election.cc:296] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 election: Requested vote from peers 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177), aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827) I0720 00:43:52.581202 3466 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "707b2290dfb44e3caa26d19101fd8b3e" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c6738598e748b383992ddac5e2c2ca" I0720 00:43:52.581267 3593 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "707b2290dfb44e3caa26d19101fd8b3e" candidate_uuid: "68ef7d6143504eeba96ee091e0d753d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" I0720 00:43:52.581328 3466 raft_consensus.cc:2890] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.581435 3593 raft_consensus.cc:2890] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:52.582864 3466 raft_consensus.cc:2350] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 1. I0720 00:43:52.582954 3593 raft_consensus.cc:2350] T 707b2290dfb44e3caa26d19101fd8b3e P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ef7d6143504eeba96ee091e0d753d3 in term 1. I0720 00:43:52.583201 3651 leader_election.cc:310] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:52.583467 3955 raft_consensus.cc:2644] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:52.583593 3955 raft_consensus.cc:667] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [term 1 LEADER]: Becoming Leader. State: Replica: 68ef7d6143504eeba96ee091e0d753d3, State: Running, Role: LEADER I0720 00:43:52.583793 3955 consensus_queue.cc:206] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.586042 3341 catalog_manager.cc:4067] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 reported cstate change: term changed from 0 to 1, leader changed from to 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131). New cstate: current_term: 1 leader_uuid: "68ef7d6143504eeba96ee091e0d753d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: HEALTHY } } } I0720 00:43:52.625602 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 3376 I0720 00:43:52.626253 3506 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:52.685453 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 3376 I0720 00:43:52.696418 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4 --block_manager=log --rpc_bind_addresses=127.0.58.129:39177 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_port=42651 --webserver_interface=127.0.58.129 --tserver_master_addrs=127.0.58.190:34297 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** --env_inject_eio=1.0 --crash_on_eio=false --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:52.847201 3974 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:52.847442 3974 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:52.849483 3974 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0720 00:43:52.849546 3974 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:52.849640 3974 flags.cc:404] Enabled experimental flag: --crash_on_eio=false W0720 00:43:52.849714 3974 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:43:52.853613 3974 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:39177 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=42651 --tserver_master_addrs=127.0.58.190:34297 --env_inject_eio=1 --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:52.861217 3974 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 77516us W0720 00:43:52.861506 3974 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:43:52.861559 3974 fault_injection.cc:45] THIS SERVER MAY CRASH! I0720 00:43:52.862190 3974 fs_manager.cc:263] Metadata directory not provided I0720 00:43:52.862284 3974 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory E0720 00:43:52.863653 3974 env_posix.cc:316] I/O error, context: INJECTED FAILURE E0720 00:43:52.865944 3974 env_posix.cc:316] I/O error, context: INJECTED FAILURE I0720 00:43:52.866019 3974 block_manager_util.cc:139] Instance is unhealthy: IO error: Failed to read metadata file from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/data/block_manager_instance: INJECTED FAILURE: Input/output error (error 5) I0720 00:43:52.873104 3974 fs_manager.cc:399] Time spent opening directory manager: real 0.008s user 0.008s sys 0.001s I0720 00:43:52.874017 3974 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:52.874541 3974 file_cache.cc:465] Constructed file cache lbm with capacity 419430 E0720 00:43:52.877918 3974 log_block_manager.cc:2011] Not using report from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5) I0720 00:43:52.878134 3974 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:52.878201 3974 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal uuid: "16c6738598e748b383992ddac5e2c2ca" format_stamp: "Formatted at 2019-07-20 00:43:51 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:52.879096 3974 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal 4 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-2/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-3/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-4/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:52.884230 3953 consensus_queue.cc:984] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:52.892117 3951 consensus_queue.cc:984] T 6042d34aeef84544bb6535ed8ca52ac3 P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:52.915465 3974 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:52.917253 3974 ts_tablet_manager.cc:345] Loading tablet metadata (0/4 complete) I0720 00:43:52.925726 3974 ts_tablet_manager.cc:357] Loaded tablet metadata (4 total tablets, 4 live tablets) I0720 00:43:52.925806 3974 ts_tablet_manager.cc:363] Registering tablets (0/4 complete) I0720 00:43:52.928738 3997 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:52.930429 3998 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. E0720 00:43:52.930657 3998 ts_tablet_manager.cc:1080] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:43:52.930765 3998 ts_tablet_manager.cc:1059] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:43:52.930874 3998 tablet_replica.cc:273] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: stopping tablet replica I0720 00:43:52.931005 3998 raft_consensus.cc:2147] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:43:52.932574 3998 raft_consensus.cc:2174] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:43:52.932929 3998 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:52.936688 3974 ts_tablet_manager.cc:377] Registered 4 tablets I0720 00:43:52.936980 3999 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. E0720 00:43:52.937191 3999 ts_tablet_manager.cc:1080] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:43:52.937358 3999 ts_tablet_manager.cc:1059] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:43:52.937465 3999 tablet_replica.cc:273] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: stopping tablet replica I0720 00:43:52.937598 3999 raft_consensus.cc:2147] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:43:52.937728 3999 raft_consensus.cc:2174] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:43:52.944090 3997 log.cc:525] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Log is configured to *not* fsync() on all Append() calls I0720 00:43:52.948058 3998 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:43:52.948164 3997 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0720 00:43:52.949081 3998 tablet_bootstrap.cc:439] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Bootstrap complete. I0720 00:43:52.949156 3997 tablet_bootstrap.cc:439] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap complete. I0720 00:43:52.950011 3997 ts_tablet_manager.cc:1059] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.014s I0720 00:43:52.950698 3998 ts_tablet_manager.cc:1059] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.006s I0720 00:43:52.963214 3997 raft_consensus.cc:340] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.963224 3998 raft_consensus.cc:340] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.966926 3998 raft_consensus.cc:705] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: FOLLOWER I0720 00:43:52.967156 3997 raft_consensus.cc:705] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: FOLLOWER I0720 00:43:52.967777 3997 consensus_queue.cc:229] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:52.967778 3998 consensus_queue.cc:229] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:52.969924 3998 ts_tablet_manager.cc:1087] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.019s user 0.012s sys 0.005s I0720 00:43:52.970010 3997 ts_tablet_manager.cc:1087] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.020s user 0.008s sys 0.011s W0720 00:43:52.979562 3651 consensus_peers.cc:458] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 -> Peer 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): Couldn't send request to peer 16c6738598e748b383992ddac5e2c2ca. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:39177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:43:52.984416 3974 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:39177 I0720 00:43:52.985430 3974 webserver.cc:374] Webserver started at http://127.0.58.129:42651/ using document root and password file I0720 00:43:52.986140 3974 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb W0720 00:43:52.987279 4109 tablet.cc:1799] T 707b2290dfb44e3caa26d19101fd8b3e P 16c6738598e748b383992ddac5e2c2ca: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:43:52.993264 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 3974 I0720 00:43:52.995470 4108 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34297 I0720 00:43:52.995663 4108 heartbeater.cc:405] Registering TS with master... I0720 00:43:52.996150 4108 heartbeater.cc:454] Master 127.0.58.190:34297 requested a full tablet report, sending... I0720 00:43:52.998201 3340 ts_manager.cc:161] Re-registered known tserver with Master: 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) W0720 00:43:52.998598 3340 catalog_manager.cc:3988] Tablet 983da8448c254fb690777c7dbccbf7fa (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) has failed on TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): IO error: some tablet data is in a failed directory W0720 00:43:52.998725 3340 catalog_manager.cc:3988] Tablet c137ad158a18463694d366278635f9b4 (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) has failed on TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): IO error: some tablet data is in a failed directory I0720 00:43:52.999665 3340 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:35815 I0720 00:43:53.051579 3525 consensus_queue.cc:544] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: The tablet replica hosted on peer 16c6738598e748b383992ddac5e2c2ca has failed W0720 00:43:53.054755 3525 consensus_peers.cc:458] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 -> Peer 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): Couldn't send request to peer 16c6738598e748b383992ddac5e2c2ca. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:43:53.074339 3952 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:53.075654 3955 consensus_queue.cc:984] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:53.076797 3955 consensus_queue.cc:984] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:53.078339 4108 heartbeater.cc:446] Master 127.0.58.190:34297 was elected leader, sending a full tablet report... W0720 00:43:53.083822 3340 catalog_manager.cc:3988] Tablet 983da8448c254fb690777c7dbccbf7fa (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) has failed on TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): IO error: some tablet data is in a failed directory W0720 00:43:53.084010 3340 catalog_manager.cc:3988] Tablet c137ad158a18463694d366278635f9b4 (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) has failed on TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): IO error: some tablet data is in a failed directory Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 456a7f6ee6f54d268c1570b10309165e | 127.0.58.190:34297 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 16c6738598e748b383992ddac5e2c2ca | 127.0.58.129:39177 | HEALTHY | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | HEALTHY | 68ef7d6143504eeba96ee091e0d753d3 | 127.0.58.131:43161 | HEALTHY | aa238bcb02fb407ebb4fdca6d3865dd4 | 127.0.58.130:46827 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:39177 env_inject_eio | 1 | hidden | 127.0.58.129:39177 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:39177 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:39177 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:46827 I0720 00:43:53.088304 3341 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 983da8448c254fb690777c7dbccbf7fa (attempt 1) local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:43161 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:34605 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:39177 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:46827 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:43161 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:34605 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 983da8448c254fb690777c7dbccbf7fa of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827): RUNNING [LEADER] All reported replicas are: A = 16c6738598e748b383992ddac5e2c2ca B = 5af276af9d674086b48b969af0dff3fc C = aa238bcb02fb407ebb4fdca6d3865dd4 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 707b2290dfb44e3caa26d19101fd8b3e of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827): RUNNING 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING [LEADER] All reported replicas are: A = 16c6738598e748b383992ddac5e2c2ca B = aa238bcb02fb407ebb4fdca6d3865dd4 C = 68ef7d6143504eeba96ee091e0d753d3 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 7548db0376b04d4b8ca358bcddb84fd5 of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [LEADER] All reported replicas are: A = 68ef7d6143504eeba96ee091e0d753d3 B = 5af276af9d674086b48b969af0dff3fc C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet c137ad158a18463694d366278635f9b4 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827): RUNNING 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING [LEADER] 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory All reported replicas are: A = aa238bcb02fb407ebb4fdca6d3865dd4 B = 68ef7d6143504eeba96ee091e0d753d3 C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | [config not available] | | | Summary by table I0720 00:43:53.090803 3593 consensus_queue.cc:206] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 1 | 0 | 2 | 2 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:43:53.093734 3844 raft_consensus.cc:1184] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Refusing update from remote peer aa238bcb02fb407ebb4fdca6d3865dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:53.094295 3953 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:43:53.097137 3953 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } } I0720 00:43:53.097609 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:43:53.100903 3844 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } } I0720 00:43:53.101332 3323 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 983da8448c254fb690777c7dbccbf7fa with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:43:53.101531 4146 consensus_queue.cc:984] T 707b2290dfb44e3caa26d19101fd8b3e P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:53.102519 3342 catalog_manager.cc:4067] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 reported cstate change: config changed from index -1 to 2, VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } } I0720 00:43:53.113059 3342 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 983da8448c254fb690777c7dbccbf7fa on 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) (TS 16c6738598e748b383992ddac5e2c2ca not found in new config with opid_index 2) I0720 00:43:53.113379 3342 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 983da8448c254fb690777c7dbccbf7fa (attempt 1) I0720 00:43:53.114709 3593 consensus_queue.cc:206] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.116267 3844 raft_consensus.cc:1184] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Refusing update from remote peer aa238bcb02fb407ebb4fdca6d3865dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:43:53.116573 3525 consensus_queue.cc:544] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: The tablet replica hosted on peer 16c6738598e748b383992ddac5e2c2ca has failed W0720 00:43:53.116689 3525 consensus_peers.cc:458] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 -> Peer 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): Couldn't send request to peer 16c6738598e748b383992ddac5e2c2ca. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:43:53.116904 3953 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:43:53.118840 3952 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } I0720 00:43:53.119482 3844 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } I0720 00:43:53.121829 3323 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 983da8448c254fb690777c7dbccbf7fa with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:43:53.122658 3342 catalog_manager.cc:4067] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 reported cstate change: config changed from index 2 to 3, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } health_report { overall_health: FAILED_UNRECOVERABLE } } } I0720 00:43:53.125603 3342 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 983da8448c254fb690777c7dbccbf7fa (attempt 1) I0720 00:43:53.126576 4045 tablet_service.cc:1066] Processing DeleteTablet for tablet 983da8448c254fb690777c7dbccbf7fa with delete_type TABLET_DATA_TOMBSTONED (TS 16c6738598e748b383992ddac5e2c2ca not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:33160 I0720 00:43:53.126842 3593 consensus_queue.cc:206] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } I0720 00:43:53.127548 4154 ts_tablet_manager.cc:1400] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:43:53.128418 3844 raft_consensus.cc:1184] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Refusing update from remote peer aa238bcb02fb407ebb4fdca6d3865dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:43:53.128924 3953 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:43:53.130822 3951 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } } I0720 00:43:53.131510 3844 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } } I0720 00:43:53.133718 3323 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 983da8448c254fb690777c7dbccbf7fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:43:53.134112 3341 catalog_manager.cc:4067] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 reported cstate change: config changed from index 3 to 4, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } } I0720 00:43:53.135526 4154 ts_tablet_manager.cc:1413] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:43:53.135637 4154 log.cc:990] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal/wals/983da8448c254fb690777c7dbccbf7fa I0720 00:43:53.136523 3324 catalog_manager.cc:3437] TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): tablet 983da8448c254fb690777c7dbccbf7fa (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) successfully deleted I0720 00:43:53.136821 3341 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 983da8448c254fb690777c7dbccbf7fa on 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) (TS 16c6738598e748b383992ddac5e2c2ca not found in new config with opid_index 4) I0720 00:43:53.137008 3341 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 983da8448c254fb690777c7dbccbf7fa (attempt 1) I0720 00:43:53.137329 4045 tablet_service.cc:1066] Processing DeleteTablet for tablet 983da8448c254fb690777c7dbccbf7fa with delete_type TABLET_DATA_TOMBSTONED (TS 16c6738598e748b383992ddac5e2c2ca not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33160 I0720 00:43:53.137902 3324 catalog_manager.cc:3437] TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): tablet 983da8448c254fb690777c7dbccbf7fa (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) successfully deleted I0720 00:43:53.138180 3593 consensus_queue.cc:206] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.139700 3844 raft_consensus.cc:1184] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Refusing update from remote peer aa238bcb02fb407ebb4fdca6d3865dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) W0720 00:43:53.139842 3525 consensus_peers.cc:458] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 -> Peer 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): Couldn't send request to peer 16c6738598e748b383992ddac5e2c2ca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:43:53.140100 3953 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0720 00:43:53.141788 3951 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } I0720 00:43:53.142323 3844 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } I0720 00:43:53.144496 3323 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 983da8448c254fb690777c7dbccbf7fa with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:43:53.145936 3342 catalog_manager.cc:4067] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc reported cstate change: config changed from index 4 to 5, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 456a7f6ee6f54d268c1570b10309165e | 127.0.58.190:34297 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 16c6738598e748b383992ddac5e2c2ca | 127.0.58.129:39177 | HEALTHY | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | HEALTHY | 68ef7d6143504eeba96ee091e0d753d3 | 127.0.58.131:43161 | HEALTHY | aa238bcb02fb407ebb4fdca6d3865dd4 | 127.0.58.130:46827 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:39177 env_inject_eio | 1 | hidden | 127.0.58.129:39177 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:39177 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:39177 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:46827 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:43161 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:34605 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:39177 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:46827 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:43161 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:34605 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 7548db0376b04d4b8ca358bcddb84fd5 of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [LEADER] All reported replicas are: A = 68ef7d6143504eeba96ee091e0d753d3 B = 5af276af9d674086b48b969af0dff3fc C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet c137ad158a18463694d366278635f9b4 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827): RUNNING 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING [LEADER] 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory All reported replicas are: A = aa238bcb02fb407ebb4fdca6d3865dd4 B = 68ef7d6143504eeba96ee091e0d753d3 C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | A B* C | 1 | -1 | Yes C | [config not available] | | | Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 3 | 0 | 1 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:43:53.317255 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:43:53.381649 3651 consensus_queue.cc:544] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: The tablet replica hosted on peer 16c6738598e748b383992ddac5e2c2ca has failed I0720 00:43:53.383625 3341 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet c137ad158a18463694d366278635f9b4 (attempt 1) I0720 00:43:53.385208 3719 consensus_queue.cc:206] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } I0720 00:43:53.386854 3593 raft_consensus.cc:1184] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Refusing update from remote peer 68ef7d6143504eeba96ee091e0d753d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:53.387403 4146 consensus_queue.cc:984] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0720 00:43:53.389495 4148 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } } I0720 00:43:53.389942 3593 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } } I0720 00:43:53.392336 3325 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet c137ad158a18463694d366278635f9b4 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:43:53.393353 3341 catalog_manager.cc:4067] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 reported cstate change: config changed from index -1 to 2, VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "68ef7d6143504eeba96ee091e0d753d3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: HEALTHY } } } I0720 00:43:53.395907 3341 catalog_manager.cc:3459] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet c137ad158a18463694d366278635f9b4 on 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177) (TS 16c6738598e748b383992ddac5e2c2ca not found in new config with opid_index 2) I0720 00:43:53.396122 3341 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c137ad158a18463694d366278635f9b4 (attempt 1) I0720 00:43:53.396585 4045 tablet_service.cc:1066] Processing DeleteTablet for tablet c137ad158a18463694d366278635f9b4 with delete_type TABLET_DATA_TOMBSTONED (TS 16c6738598e748b383992ddac5e2c2ca not found in new config with opid_index 2) from {username='slave'} at 127.0.0.1:33160 I0720 00:43:53.396886 4154 ts_tablet_manager.cc:1400] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0720 00:43:53.397624 3719 consensus_queue.cc:206] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.398574 3341 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c137ad158a18463694d366278635f9b4 (attempt 1) I0720 00:43:53.399192 3593 raft_consensus.cc:1184] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Refusing update from remote peer 68ef7d6143504eeba96ee091e0d753d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) W0720 00:43:53.399605 3651 consensus_peers.cc:458] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 -> Peer 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): Couldn't send request to peer 16c6738598e748b383992ddac5e2c2ca. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: FAILED. This is attempt 1: this message will repeat every 5th retry. I0720 00:43:53.399647 4146 consensus_queue.cc:984] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0720 00:43:53.400833 3325 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c137ad158a18463694d366278635f9b4 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } }. Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.400969 3325 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c137ad158a18463694d366278635f9b4 with cas_config_opid_index 2 with a delay of 32 ms (attempt = 1) I0720 00:43:53.401441 4154 ts_tablet_manager.cc:1413] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId unknown I0720 00:43:53.401526 4154 log.cc:990] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Deleting WAL directory at /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/wal/wals/c137ad158a18463694d366278635f9b4 I0720 00:43:53.401721 4148 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } I0720 00:43:53.402199 3324 catalog_manager.cc:3437] TS 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): tablet c137ad158a18463694d366278635f9b4 (table test-workload [id=e18b4d8e3ba84cfa8b5d1554bb968ede]) successfully deleted I0720 00:43:53.402318 3593 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } } I0720 00:43:53.403889 3325 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c137ad158a18463694d366278635f9b4 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:43:53.405194 3341 catalog_manager.cc:4067] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 reported cstate change: config changed from index 2 to 3, NON_VOTER 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) added. New cstate: current_term: 1 leader_uuid: "68ef7d6143504eeba96ee091e0d753d3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:43:53.433581 3325 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c137ad158a18463694d366278635f9b4 with cas_config_opid_index 2: aborting the task: latest config opid_index 3; task opid_index 2 Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 456a7f6ee6f54d268c1570b10309165e | 127.0.58.190:34297 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 16c6738598e748b383992ddac5e2c2ca | 127.0.58.129:39177 | HEALTHY | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | HEALTHY | 68ef7d6143504eeba96ee091e0d753d3 | 127.0.58.131:43161 | HEALTHY | aa238bcb02fb407ebb4fdca6d3865dd4 | 127.0.58.130:46827 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:39177 env_inject_eio | 1 | hidden | 127.0.58.129:39177 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:39177 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:39177 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:46827 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:43161 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:34605 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:39177 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:46827 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:43161 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:34605 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 7548db0376b04d4b8ca358bcddb84fd5 of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [LEADER] All reported replicas are: A = 68ef7d6143504eeba96ee091e0d753d3 B = 5af276af9d674086b48b969af0dff3fc C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 4 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:43:53.609673 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:43:53.701483 4202 ts_tablet_manager.cc:682] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Initiating tablet copy from peer aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827) I0720 00:43:53.701692 4202 tablet_copy_client.cc:204] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:43:53.701841 4202 tablet_copy_client.cc:241] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.130:46827 I0720 00:43:53.707402 3613 tablet_copy_service.cc:135] P aa238bcb02fb407ebb4fdca6d3865dd4: Received BeginTabletCopySession request for tablet 983da8448c254fb690777c7dbccbf7fa from peer 16c6738598e748b383992ddac5e2c2ca ({username='slave'} at 127.0.58.129:38179) I0720 00:43:53.707587 3613 tablet_copy_service.cc:156] P aa238bcb02fb407ebb4fdca6d3865dd4: Beginning new tablet copy session on tablet 983da8448c254fb690777c7dbccbf7fa from peer 16c6738598e748b383992ddac5e2c2ca at {username='slave'} at 127.0.58.129:38179: session id = 16c6738598e748b383992ddac5e2c2ca-983da8448c254fb690777c7dbccbf7fa I0720 00:43:53.709684 3613 tablet_copy_source_session.cc:202] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:43:53.710981 4202 ts_tablet_manager.cc:1400] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:43:53.714881 4202 ts_tablet_manager.cc:1413] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:43:53.717218 4202 tablet_copy_client.cc:579] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Starting download of 0 data blocks... I0720 00:43:53.717425 4202 tablet_copy_client.cc:542] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Starting download of 1 WAL segments... I0720 00:43:53.719108 4202 tablet_copy_client.cc:418] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:43:53.721066 4202 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:53.727068 4202 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:43:53.727425 4202 tablet_bootstrap.cc:439] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Bootstrap complete. I0720 00:43:53.727581 4202 ts_tablet_manager.cc:1059] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s I0720 00:43:53.728144 4202 raft_consensus.cc:340] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.728325 4202 raft_consensus.cc:705] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: LEARNER I0720 00:43:53.728488 4202 consensus_queue.cc:229] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.729044 4202 ts_tablet_manager.cc:1087] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s I0720 00:43:53.729903 3613 tablet_copy_service.cc:325] P aa238bcb02fb407ebb4fdca6d3865dd4: Request end of tablet copy session 16c6738598e748b383992ddac5e2c2ca-983da8448c254fb690777c7dbccbf7fa received from {username='slave'} at 127.0.58.129:38179 I0720 00:43:53.730037 3613 tablet_copy_service.cc:413] P aa238bcb02fb407ebb4fdca6d3865dd4: ending tablet copy session 16c6738598e748b383992ddac5e2c2ca-983da8448c254fb690777c7dbccbf7fa on tablet 983da8448c254fb690777c7dbccbf7fa with peer 16c6738598e748b383992ddac5e2c2ca I0720 00:43:53.965245 4202 ts_tablet_manager.cc:682] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Initiating tablet copy from peer 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161) I0720 00:43:53.965344 4202 tablet_copy_client.cc:204] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet copy: overwriting existing tombstoned replica with an unknown last-logged opid I0720 00:43:53.965487 4202 tablet_copy_client.cc:241] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Beginning tablet copy session from remote peer at address 127.0.58.131:43161 I0720 00:43:53.971367 3739 tablet_copy_service.cc:135] P 68ef7d6143504eeba96ee091e0d753d3: Received BeginTabletCopySession request for tablet c137ad158a18463694d366278635f9b4 from peer 16c6738598e748b383992ddac5e2c2ca ({username='slave'} at 127.0.58.129:54999) I0720 00:43:53.971586 3739 tablet_copy_service.cc:156] P 68ef7d6143504eeba96ee091e0d753d3: Beginning new tablet copy session on tablet c137ad158a18463694d366278635f9b4 from peer 16c6738598e748b383992ddac5e2c2ca at {username='slave'} at 127.0.58.129:54999: session id = 16c6738598e748b383992ddac5e2c2ca-c137ad158a18463694d366278635f9b4 I0720 00:43:53.975119 3739 tablet_copy_source_session.cc:202] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: Tablet Copy: opened 0 blocks and 1 log segments I0720 00:43:53.976446 4202 ts_tablet_manager.cc:1400] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Deleting tablet data with delete state TABLET_DATA_COPYING I0720 00:43:53.980355 4202 ts_tablet_manager.cc:1413] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId unknown I0720 00:43:53.982765 4202 tablet_copy_client.cc:579] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Starting download of 0 data blocks... I0720 00:43:53.982980 4202 tablet_copy_client.cc:542] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Starting download of 1 WAL segments... I0720 00:43:53.984700 4202 tablet_copy_client.cc:418] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0720 00:43:53.988620 4202 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap starting. I0720 00:43:53.994067 4202 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0720 00:43:53.994479 4202 tablet_bootstrap.cc:439] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Bootstrap complete. I0720 00:43:53.994675 4202 ts_tablet_manager.cc:1059] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.005s I0720 00:43:53.995420 4202 raft_consensus.cc:340] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:53.995687 4202 raft_consensus.cc:705] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Initialized, Role: LEARNER I0720 00:43:53.995925 4202 consensus_queue.cc:229] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: NON_VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: true } } I0720 00:43:54.000552 4202 ts_tablet_manager.cc:1087] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s I0720 00:43:54.001238 3739 tablet_copy_service.cc:325] P 68ef7d6143504eeba96ee091e0d753d3: Request end of tablet copy session 16c6738598e748b383992ddac5e2c2ca-c137ad158a18463694d366278635f9b4 received from {username='slave'} at 127.0.58.129:54999 I0720 00:43:54.001334 3739 tablet_copy_service.cc:413] P 68ef7d6143504eeba96ee091e0d753d3: ending tablet copy session 16c6738598e748b383992ddac5e2c2ca-c137ad158a18463694d366278635f9b4 on tablet c137ad158a18463694d366278635f9b4 with peer 16c6738598e748b383992ddac5e2c2ca Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 456a7f6ee6f54d268c1570b10309165e | 127.0.58.190:34297 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 16c6738598e748b383992ddac5e2c2ca | 127.0.58.129:39177 | HEALTHY | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | HEALTHY | 68ef7d6143504eeba96ee091e0d753d3 | 127.0.58.131:43161 | HEALTHY | aa238bcb02fb407ebb4fdca6d3865dd4 | 127.0.58.130:46827 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:39177 env_inject_eio | 1 | hidden | 127.0.58.129:39177 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:39177 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:39177 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:46827 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:43161 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:34605 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:39177 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:46827 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:43161 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:34605 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 983da8448c254fb690777c7dbccbf7fa of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827): RUNNING [LEADER] 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [NONVOTER] All reported replicas are: A = 5af276af9d674086b48b969af0dff3fc B = aa238bcb02fb407ebb4fdca6d3865dd4 C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 5 | Yes B | A B* C~ | 1 | 5 | Yes C | A B C~ | 1 | 5 | Yes Tablet 7548db0376b04d4b8ca358bcddb84fd5 of table 'test-workload' is conflicted: 6 replicas' active configs disagree with the leader master's 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [LEADER] All reported replicas are: A = 68ef7d6143504eeba96ee091e0d753d3 B = 5af276af9d674086b48b969af0dff3fc C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 1 | -1 | Yes B | A B C | 1 | -1 | Yes C | A B C | 1 | -1 | Yes Tablet c137ad158a18463694d366278635f9b4 of table 'test-workload' is conflicted: 1 replicas' active configs disagree with the leader master's aa238bcb02fb407ebb4fdca6d3865dd4 (127.0.58.130:46827): RUNNING 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING [LEADER] 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [NONVOTER] All reported replicas are: A = aa238bcb02fb407ebb4fdca6d3865dd4 B = 68ef7d6143504eeba96ee091e0d753d3 C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C~ | | | Yes A | A B* C~ | 1 | 3 | Yes B | A B* C~ | 1 | 3 | Yes C | A B C~ | 1 | 3 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 2 | 0 | 0 | 3 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:43:54.029546 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:43:54.216173 4205 raft_consensus.cc:472] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:54.216363 4205 raft_consensus.cc:494] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:54.217669 4205 leader_election.cc:296] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161), 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) I0720 00:43:54.218297 3719 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68ef7d6143504eeba96ee091e0d753d3" is_pre_election: true I0720 00:43:54.218539 3719 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 1. I0720 00:43:54.219072 3987 leader_election.cc:310] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:54.219444 4205 raft_consensus.cc:2644] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:43:54.219527 4205 raft_consensus.cc:472] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:54.219581 4205 raft_consensus.cc:2890] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Advancing to term 2 I0720 00:43:54.222146 4205 raft_consensus.cc:494] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:54.222620 4205 leader_election.cc:296] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 2 election: Requested vote from peers 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161), 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605) I0720 00:43:54.223124 3719 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "68ef7d6143504eeba96ee091e0d753d3" I0720 00:43:54.223279 3719 raft_consensus.cc:2890] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:43:54.223970 3844 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" is_pre_election: true I0720 00:43:54.224073 3845 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7548db0376b04d4b8ca358bcddb84fd5" candidate_uuid: "16c6738598e748b383992ddac5e2c2ca" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5af276af9d674086b48b969af0dff3fc" I0720 00:43:54.224144 3844 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 1. I0720 00:43:54.224249 3845 raft_consensus.cc:2890] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Advancing to term 2 I0720 00:43:54.225332 3719 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 2. I0720 00:43:54.225700 3987 leader_election.cc:310] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c6738598e748b383992ddac5e2c2ca, 68ef7d6143504eeba96ee091e0d753d3; no voters: I0720 00:43:54.225967 4205 raft_consensus.cc:2644] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:43:54.226316 3845 raft_consensus.cc:2350] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16c6738598e748b383992ddac5e2c2ca in term 2. I0720 00:43:54.226315 4205 raft_consensus.cc:667] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [term 2 LEADER]: Becoming Leader. State: Replica: 16c6738598e748b383992ddac5e2c2ca, State: Running, Role: LEADER I0720 00:43:54.226686 4205 consensus_queue.cc:206] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } } I0720 00:43:54.229233 3342 catalog_manager.cc:4067] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "16c6738598e748b383992ddac5e2c2ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } health_report { overall_health: HEALTHY } } } I0720 00:43:54.230808 4065 raft_consensus.cc:1126] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[] I0720 00:43:54.517240 4065 raft_consensus.cc:1126] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[] Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 456a7f6ee6f54d268c1570b10309165e | 127.0.58.190:34297 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 16c6738598e748b383992ddac5e2c2ca | 127.0.58.129:39177 | HEALTHY | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | HEALTHY | 68ef7d6143504eeba96ee091e0d753d3 | 127.0.58.131:43161 | HEALTHY | aa238bcb02fb407ebb4fdca6d3865dd4 | 127.0.58.130:46827 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:39177 env_inject_eio | 1 | hidden | 127.0.58.129:39177 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:39177 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:39177 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:46827 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:43161 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:34605 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:39177 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:46827 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:43161 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:34605 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 7548db0376b04d4b8ca358bcddb84fd5 of table 'test-workload' is conflicted: 5 replicas' active configs disagree with the leader master's 68ef7d6143504eeba96ee091e0d753d3 (127.0.58.131:43161): RUNNING 5af276af9d674086b48b969af0dff3fc (127.0.58.132:34605): RUNNING 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): RUNNING [LEADER] All reported replicas are: A = 68ef7d6143504eeba96ee091e0d753d3 B = 5af276af9d674086b48b969af0dff3fc C = 16c6738598e748b383992ddac5e2c2ca The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | A B C | 2 | -1 | Yes C | A B C* | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 5 | 4 | 0 | 0 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:43:54.600875 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:43:54.662590 3719 raft_consensus.cc:1184] T 7548db0376b04d4b8ca358bcddb84fd5 P 68ef7d6143504eeba96ee091e0d753d3 [term 2 FOLLOWER]: Refusing update from remote peer 16c6738598e748b383992ddac5e2c2ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0720 00:43:54.663187 4205 consensus_queue.cc:984] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:54.669860 3845 raft_consensus.cc:1184] T 7548db0376b04d4b8ca358bcddb84fd5 P 5af276af9d674086b48b969af0dff3fc [term 2 FOLLOWER]: Refusing update from remote peer 16c6738598e748b383992ddac5e2c2ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0720 00:43:54.670395 4205 consensus_queue.cc:984] T 7548db0376b04d4b8ca358bcddb84fd5 P 16c6738598e748b383992ddac5e2c2ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:54.731678 4234 raft_consensus.cc:974] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4: attempting to promote NON_VOTER 16c6738598e748b383992ddac5e2c2ca to VOTER I0720 00:43:54.732306 4234 consensus_queue.cc:206] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } I0720 00:43:54.734406 4065 raft_consensus.cc:1184] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Refusing update from remote peer aa238bcb02fb407ebb4fdca6d3865dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I0720 00:43:54.734428 3845 raft_consensus.cc:1184] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Refusing update from remote peer aa238bcb02fb407ebb4fdca6d3865dd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I0720 00:43:54.734969 4233 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0720 00:43:54.735275 4276 consensus_queue.cc:984] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0720 00:43:54.737730 4233 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } } I0720 00:43:54.738437 4065 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } } I0720 00:43:54.738735 3845 raft_consensus.cc:2792] T 983da8448c254fb690777c7dbccbf7fa P 5af276af9d674086b48b969af0dff3fc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } } I0720 00:43:54.741973 3342 catalog_manager.cc:4067] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 reported cstate change: config changed from index 5 to 6, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5af276af9d674086b48b969af0dff3fc" member_type: VOTER last_known_addr { host: "127.0.58.132" port: 34605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0720 00:43:54.982717 4239 raft_consensus.cc:974] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3: attempting to promote NON_VOTER 16c6738598e748b383992ddac5e2c2ca to VOTER I0720 00:43:54.983286 4239 consensus_queue.cc:206] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } I0720 00:43:54.985404 3593 raft_consensus.cc:1184] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Refusing update from remote peer 68ef7d6143504eeba96ee091e0d753d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:43:54.985558 4065 raft_consensus.cc:1184] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 LEARNER]: Refusing update from remote peer 68ef7d6143504eeba96ee091e0d753d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:43:54.986002 4238 consensus_queue.cc:984] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:43:54.986327 4239 consensus_queue.cc:984] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0720 00:43:54.989615 3593 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P aa238bcb02fb407ebb4fdca6d3865dd4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } } I0720 00:43:54.989051 3955 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } } I0720 00:43:54.990800 4065 raft_consensus.cc:2792] T c137ad158a18463694d366278635f9b4 P 16c6738598e748b383992ddac5e2c2ca [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } } } I0720 00:43:54.993850 3340 catalog_manager.cc:4067] T c137ad158a18463694d366278635f9b4 P 68ef7d6143504eeba96ee091e0d753d3 reported cstate change: config changed from index 3 to 4, 16c6738598e748b383992ddac5e2c2ca (127.0.58.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "68ef7d6143504eeba96ee091e0d753d3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "aa238bcb02fb407ebb4fdca6d3865dd4" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 46827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ef7d6143504eeba96ee091e0d753d3" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 43161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16c6738598e748b383992ddac5e2c2ca" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 39177 } attrs { promote: false } health_report { overall_health: HEALTHY } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 456a7f6ee6f54d268c1570b10309165e | 127.0.58.190:34297 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 16c6738598e748b383992ddac5e2c2ca | 127.0.58.129:39177 | HEALTHY | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | HEALTHY | 68ef7d6143504eeba96ee091e0d753d3 | 127.0.58.131:43161 | HEALTHY | aa238bcb02fb407ebb4fdca6d3865dd4 | 127.0.58.130:46827 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- crash_on_eio | false | experimental,advanced | 127.0.58.129:39177 env_inject_eio | 1 | hidden | 127.0.58.129:39177 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:39177 ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:39177 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:46827 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:43161 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:34605 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:39177 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:46827 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:43161 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskFailure_DiskFailureITest.TestFailDuringServerStartup_3.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:34605 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 5 | 5 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 3 First Quartile | 4 Median | 4 Third Quartile | 4 Maximum | 4 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 5af276af9d674086b48b969af0dff3fc | 127.0.58.132:34605 | 3 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 5 Replicas | 15 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:43:55.439991 234 log_verifier.cc:121] Checking tablet 6042d34aeef84544bb6535ed8ca52ac3 I0720 00:43:55.464228 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 6042d34aeef84544bb6535ed8ca52ac3 I0720 00:43:55.464324 234 log_verifier.cc:121] Checking tablet 707b2290dfb44e3caa26d19101fd8b3e I0720 00:43:55.489063 234 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 707b2290dfb44e3caa26d19101fd8b3e I0720 00:43:55.489148 234 log_verifier.cc:121] Checking tablet 7548db0376b04d4b8ca358bcddb84fd5 I0720 00:43:55.514719 234 log_verifier.cc:172] Verified matching terms for 2 ops in tablet 7548db0376b04d4b8ca358bcddb84fd5 I0720 00:43:55.514811 234 log_verifier.cc:121] Checking tablet 983da8448c254fb690777c7dbccbf7fa I0720 00:43:55.542171 234 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 983da8448c254fb690777c7dbccbf7fa I0720 00:43:55.542263 234 log_verifier.cc:121] Checking tablet c137ad158a18463694d366278635f9b4 I0720 00:43:55.567809 234 log_verifier.cc:172] Verified matching terms for 4 ops in tablet c137ad158a18463694d366278635f9b4 I0720 00:43:55.980926 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 3974 I0720 00:43:55.993367 4105 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:56.136829 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 3974 W0720 00:43:56.151660 3776 connection.cc:655] server connection from 127.0.58.129:57101 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:43:56.152343 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 3513 W0720 00:43:56.158720 3525 consensus_peers.cc:458] T 983da8448c254fb690777c7dbccbf7fa P aa238bcb02fb407ebb4fdca6d3865dd4 -> Peer 16c6738598e748b383992ddac5e2c2ca (127.0.58.129:39177): Couldn't send request to peer 16c6738598e748b383992ddac5e2c2ca. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:39177: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:43:56.159623 3633 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:56.226637 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 3513 W0720 00:43:56.237663 3777 connection.cc:655] server connection from 127.0.58.130:60655 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:43:56.238296 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 3639 I0720 00:43:56.243580 3759 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:56.338208 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 3639 I0720 00:43:56.353489 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 3765 I0720 00:43:56.359133 3885 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:56.435473 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 3765 I0720 00:43:56.451651 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 3312 I0720 00:43:56.452693 3372 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:43:56.535006 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 3312 [ OK ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/3 (5788 ms) [----------] 4 tests from DiskFailure/DiskFailureITest (89062 ms total) [----------] 4 tests from DiskError/DiskErrorITest [ RUN ] DiskError/DiskErrorITest.TestFailOnBootstrap/0 I0720 00:43:56.576229 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-2 --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:42541 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_tombstone_evicted_tablet_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:56.796506 4336 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:56.796743 4336 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:56.799325 4336 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:43:56.799376 4336 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:56.799410 4336 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:43:56.799454 4336 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:43:56.804591 4336 master_main.cc:98] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/wal --master_tombstone_evicted_tablet_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:42541 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:56.812258 4336 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 79516us I0720 00:43:56.813681 4336 fs_manager.cc:263] Metadata directory not provided I0720 00:43:56.813766 4336 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:56.814117 4336 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:56.814203 4336 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:56.818142 4336 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "537ffdd6efb847e991c0d499bb244e71" format_stamp: "Formatted at 2019-07-20 00:43:56 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:56.818790 4336 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "537ffdd6efb847e991c0d499bb244e71" format_stamp: "Formatted at 2019-07-20 00:43:56 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:56.819139 4336 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "537ffdd6efb847e991c0d499bb244e71" format_stamp: "Formatted at 2019-07-20 00:43:56 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:56.819465 4336 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "537ffdd6efb847e991c0d499bb244e71" format_stamp: "Formatted at 2019-07-20 00:43:56 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:56.830871 4336 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s I0720 00:43:56.832288 4336 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:56.832931 4336 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:56.836751 4336 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0720 00:43:56.836832 4336 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/wal uuid: "537ffdd6efb847e991c0d499bb244e71" format_stamp: "Formatted at 2019-07-20 00:43:56 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:56.837014 4336 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:56.873246 4336 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:56.889768 4336 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:42541 I0720 00:43:56.891141 4336 webserver.cc:374] Webserver started at http://127.0.58.190:37801/ using document root and password file I0720 00:43:56.892024 4336 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:43:56.896914 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 4336 I0720 00:43:56.897697 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:42541 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:43:56.910866 4397 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71: Bootstrap starting. I0720 00:43:56.913691 4397 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71: No blocks or log segments found. Creating new log. I0720 00:43:56.926846 4397 log.cc:525] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71: Log is configured to *not* fsync() on all Append() calls I0720 00:43:56.929268 4397 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71: No bootstrap required, opened a new log I0720 00:43:56.936431 4397 raft_consensus.cc:340] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } I0720 00:43:56.936738 4397 raft_consensus.cc:366] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:56.936846 4397 raft_consensus.cc:705] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537ffdd6efb847e991c0d499bb244e71, State: Initialized, Role: FOLLOWER I0720 00:43:56.937400 4397 consensus_queue.cc:229] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } I0720 00:43:56.937605 4397 raft_consensus.cc:380] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:43:56.937669 4397 raft_consensus.cc:472] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:43:56.937775 4397 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:56.941059 4397 raft_consensus.cc:494] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } I0720 00:43:56.941659 4397 leader_election.cc:310] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 537ffdd6efb847e991c0d499bb244e71; no voters: I0720 00:43:56.942432 4397 leader_election.cc:296] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:43:56.942787 4400 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:56.943819 4400 raft_consensus.cc:667] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [term 1 LEADER]: Becoming Leader. State: Replica: 537ffdd6efb847e991c0d499bb244e71, State: Running, Role: LEADER I0720 00:43:56.944514 4400 consensus_queue.cc:206] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } I0720 00:43:56.948323 4402 sys_catalog.cc:340] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 537ffdd6efb847e991c0d499bb244e71. Latest consensus state: current_term: 1 leader_uuid: "537ffdd6efb847e991c0d499bb244e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } } I0720 00:43:56.948725 4402 sys_catalog.cc:343] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: This master's current role is: LEADER I0720 00:43:56.948958 4397 sys_catalog.cc:340] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "537ffdd6efb847e991c0d499bb244e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } } I0720 00:43:56.949435 4397 sys_catalog.cc:343] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: This master's current role is: LEADER I0720 00:43:56.949273 4401 sys_catalog.cc:340] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "537ffdd6efb847e991c0d499bb244e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "537ffdd6efb847e991c0d499bb244e71" member_type: VOTER } } I0720 00:43:56.949643 4401 sys_catalog.cc:343] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: This master's current role is: LEADER I0720 00:43:56.950944 4397 sys_catalog.cc:428] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71 [sys.catalog]: configured and running, proceeding with master startup. I0720 00:43:56.951081 4406 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:43:56.959990 4406 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:57.065933 4398 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:57.066236 4398 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:57.068264 4398 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:57.068357 4398 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:43:57.068401 4398 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:43:57.068449 4398 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:43:57.094292 4398 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:42541 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:57.101898 4398 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 80016us I0720 00:43:57.102584 4398 fs_manager.cc:263] Metadata directory not provided I0720 00:43:57.102663 4398 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:57.102928 4398 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:57.103003 4398 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:57.105242 4398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "506b8430b3284890b468e7ef55b6ca79" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.105741 4398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "506b8430b3284890b468e7ef55b6ca79" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.106065 4398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "506b8430b3284890b468e7ef55b6ca79" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.106402 4398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "506b8430b3284890b468e7ef55b6ca79" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.124049 4406 catalog_manager.cc:959] Generated new certificate authority record I0720 00:43:57.125463 4406 catalog_manager.cc:1084] Loading token signing keys... I0720 00:43:57.131948 4398 fs_manager.cc:503] Time spent creating directory manager: real 0.025s user 0.014s sys 0.011s I0720 00:43:57.133316 4398 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:57.133846 4398 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:57.136098 4406 catalog_manager.cc:4371] T 00000000000000000000000000000000 P 537ffdd6efb847e991c0d499bb244e71: Generated new TSK 0 I0720 00:43:57.137802 4398 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s I0720 00:43:57.137873 4398 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal uuid: "506b8430b3284890b468e7ef55b6ca79" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.138025 4398 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:57.186841 4398 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:57.189030 4398 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:57.189098 4398 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:57.332180 4398 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:36375 I0720 00:43:57.333525 4398 webserver.cc:374] Webserver started at http://127.0.58.129:33417/ using document root and password file I0720 00:43:57.334475 4398 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:43:57.343261 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 4398 I0720 00:43:57.344008 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:42541 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:43:57.351402 4530 heartbeater.cc:324] Connected to a master server at 127.0.58.190:42541 I0720 00:43:57.351626 4530 heartbeater.cc:405] Registering TS with master... I0720 00:43:57.352121 4530 heartbeater.cc:454] Master 127.0.58.190:42541 requested a full tablet report, sending... I0720 00:43:57.354185 4364 ts_manager.cc:161] Registered new tserver with Master: 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) I0720 00:43:57.355886 4364 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:55141 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:57.521333 4534 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:57.521615 4534 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:57.523802 4534 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:57.523897 4534 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:43:57.523947 4534 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:43:57.523999 4534 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:43:57.528381 4534 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:42541 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:57.537714 4534 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 80016us I0720 00:43:57.538523 4534 fs_manager.cc:263] Metadata directory not provided I0720 00:43:57.538605 4534 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:57.538911 4534 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:57.538997 4534 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:57.541635 4534 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "a29a463e4c2e4f61a400dde426c625e8" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.542259 4534 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "a29a463e4c2e4f61a400dde426c625e8" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.542627 4534 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "a29a463e4c2e4f61a400dde426c625e8" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.542940 4534 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "a29a463e4c2e4f61a400dde426c625e8" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.552429 4534 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s I0720 00:43:57.553866 4534 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:57.554553 4534 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:57.557992 4534 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:43:57.558064 4534 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/wal uuid: "a29a463e4c2e4f61a400dde426c625e8" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.558274 4534 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:57.591154 4534 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:57.593039 4534 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:57.593112 4534 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:57.621623 4534 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:43527 I0720 00:43:57.622855 4534 webserver.cc:374] Webserver started at http://127.0.58.130:43579/ using document root and password file I0720 00:43:57.623592 4534 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:43:57.633949 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 4534 I0720 00:43:57.634711 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:42541 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:43:57.637936 4655 heartbeater.cc:324] Connected to a master server at 127.0.58.190:42541 I0720 00:43:57.638231 4655 heartbeater.cc:405] Registering TS with master... I0720 00:43:57.638782 4655 heartbeater.cc:454] Master 127.0.58.190:42541 requested a full tablet report, sending... I0720 00:43:57.640053 4364 ts_manager.cc:161] Registered new tserver with Master: a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:57.640923 4364 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:35167 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:43:57.818608 4658 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:43:57.818869 4658 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:43:57.821084 4658 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:43:57.821183 4658 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:43:57.821226 4658 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:43:57.821285 4658 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:43:57.825711 4658 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:42541 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:43:57.834899 4658 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 80016us I0720 00:43:57.835677 4658 fs_manager.cc:263] Metadata directory not provided I0720 00:43:57.835755 4658 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:43:57.836073 4658 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:43:57.836158 4658 server_base.cc:449] Attempting to create new FS layout instead I0720 00:43:57.838836 4658 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "fc86e9a330584f00b998288875c0586f" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.839431 4658 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "fc86e9a330584f00b998288875c0586f" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.839771 4658 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "fc86e9a330584f00b998288875c0586f" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.840087 4658 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "fc86e9a330584f00b998288875c0586f" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.850019 4658 fs_manager.cc:503] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s I0720 00:43:57.851445 4658 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:43:57.852041 4658 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:43:57.858810 4658 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s I0720 00:43:57.858897 4658 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/wal uuid: "fc86e9a330584f00b998288875c0586f" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:43:57.859097 4658 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:43:57.918987 4658 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:43:57.920871 4658 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:43:57.920943 4658 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:43:57.950635 4658 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:42343 I0720 00:43:57.951865 4658 webserver.cc:374] Webserver started at http://127.0.58.131:34477/ using document root and password file I0720 00:43:57.952581 4658 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:43:57.955581 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 4658 I0720 00:43:57.968307 4779 heartbeater.cc:324] Connected to a master server at 127.0.58.190:42541 I0720 00:43:57.968533 4779 heartbeater.cc:405] Registering TS with master... I0720 00:43:57.969048 4779 heartbeater.cc:454] Master 127.0.58.190:42541 requested a full tablet report, sending... I0720 00:43:57.970274 4364 ts_manager.cc:161] Registered new tserver with Master: fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:57.971195 4364 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:53031 I0720 00:43:57.971722 234 external_mini_cluster.cc:628] 3 TS(s) registered with all masters I0720 00:43:57.983539 234 test_util.cc:212] Using random seed: 593874719 I0720 00:43:57.999691 4364 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41892: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s" } partition_schema { range_schema { columns { name: "key" } } } W0720 00:43:58.001680 4364 catalog_manager.cc:1583] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0720 00:43:58.031724 4716 tablet_service.cc:1022] Processing CreateTablet for tablet 73ac7512c8584fe3ad887ffef570df8d (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:43:58.035213 4589 tablet_service.cc:1022] Processing CreateTablet for tablet 27285c1e3c36466f9bce8450be7500d4 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:43:58.035799 4466 tablet_service.cc:1022] Processing CreateTablet for tablet 5758df4920ce4bbab0424048f13fea41 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:43:58.036619 4464 tablet_service.cc:1022] Processing CreateTablet for tablet 27285c1e3c36466f9bce8450be7500d4 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:43:58.036309 4459 tablet_service.cc:1022] Processing CreateTablet for tablet 7869d31b2ece4c38b8cf73ba177d2787 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:43:58.036413 4460 tablet_service.cc:1022] Processing CreateTablet for tablet 0a7ad36f80f44067ba0087a2768e26ac (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:43:58.036017 4461 tablet_service.cc:1022] Processing CreateTablet for tablet ca6a2a19cbbe4648a8a76de1e43c405b (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:43:58.036938 4463 tablet_service.cc:1022] Processing CreateTablet for tablet 73ea05088cb24563bcf684d6199dd946 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:43:58.036571 4465 tablet_service.cc:1022] Processing CreateTablet for tablet 9ef65edd881743f691ebb903dad80c36 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:43:58.037250 4462 tablet_service.cc:1022] Processing CreateTablet for tablet 19920b309dd84a01b2cc74ddbbd10854 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:43:58.037508 4467 tablet_service.cc:1022] Processing CreateTablet for tablet 73ac7512c8584fe3ad887ffef570df8d (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:43:58.038316 4586 tablet_service.cc:1022] Processing CreateTablet for tablet ca6a2a19cbbe4648a8a76de1e43c405b (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:43:58.038604 4715 tablet_service.cc:1022] Processing CreateTablet for tablet 5758df4920ce4bbab0424048f13fea41 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:43:58.039057 4585 tablet_service.cc:1022] Processing CreateTablet for tablet 0a7ad36f80f44067ba0087a2768e26ac (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:43:58.039038 4458 tablet_service.cc:1022] Processing CreateTablet for tablet c4890cfc794b469d8765701546219a19 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:43:58.039746 4590 tablet_service.cc:1022] Processing CreateTablet for tablet 9ef65edd881743f691ebb903dad80c36 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:43:58.040405 4588 tablet_service.cc:1022] Processing CreateTablet for tablet 73ea05088cb24563bcf684d6199dd946 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:43:58.041303 4591 tablet_service.cc:1022] Processing CreateTablet for tablet 5758df4920ce4bbab0424048f13fea41 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:43:58.042214 4584 tablet_service.cc:1022] Processing CreateTablet for tablet 7869d31b2ece4c38b8cf73ba177d2787 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:43:58.042757 4583 tablet_service.cc:1022] Processing CreateTablet for tablet c4890cfc794b469d8765701546219a19 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:43:58.043321 4587 tablet_service.cc:1022] Processing CreateTablet for tablet 19920b309dd84a01b2cc74ddbbd10854 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:43:58.043949 4592 tablet_service.cc:1022] Processing CreateTablet for tablet 73ac7512c8584fe3ad887ffef570df8d (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:43:58.045395 4714 tablet_service.cc:1022] Processing CreateTablet for tablet 9ef65edd881743f691ebb903dad80c36 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:43:58.046332 4712 tablet_service.cc:1022] Processing CreateTablet for tablet 73ea05088cb24563bcf684d6199dd946 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:43:58.049393 4711 tablet_service.cc:1022] Processing CreateTablet for tablet ca6a2a19cbbe4648a8a76de1e43c405b (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:43:58.049675 4713 tablet_service.cc:1022] Processing CreateTablet for tablet 27285c1e3c36466f9bce8450be7500d4 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:43:58.051703 4715 tablet_service.cc:1022] Processing CreateTablet for tablet 19920b309dd84a01b2cc74ddbbd10854 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:43:58.052700 4710 tablet_service.cc:1022] Processing CreateTablet for tablet 0a7ad36f80f44067ba0087a2768e26ac (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:43:58.054715 4709 tablet_service.cc:1022] Processing CreateTablet for tablet 7869d31b2ece4c38b8cf73ba177d2787 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:43:58.055876 4708 tablet_service.cc:1022] Processing CreateTablet for tablet c4890cfc794b469d8765701546219a19 (table=test-workload [id=03430f12567b4e1ab531baa411322323]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:43:58.087625 4803 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.088268 4804 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.091869 4804 tablet_bootstrap.cc:590] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.093526 4804 log.cc:525] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f: Log is configured to *not* fsync() on all Append() calls I0720 00:43:58.092664 4803 tablet_bootstrap.cc:590] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.088443 4805 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.095122 4805 tablet_bootstrap.cc:590] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.111271 4803 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.111276 4804 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.111930 4803 ts_tablet_manager.cc:1059] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s I0720 00:43:58.113688 4804 ts_tablet_manager.cc:1059] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s I0720 00:43:58.114118 4805 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.118376 4805 ts_tablet_manager.cc:1059] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s I0720 00:43:58.119278 4810 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.119544 4805 raft_consensus.cc:340] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.120124 4805 raft_consensus.cc:366] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.120227 4805 raft_consensus.cc:705] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.120834 4805 consensus_queue.cc:229] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.121999 4804 raft_consensus.cc:340] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.122900 4779 heartbeater.cc:446] Master 127.0.58.190:42541 was elected leader, sending a full tablet report... I0720 00:43:58.122973 4813 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.123565 4804 raft_consensus.cc:366] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.123759 4804 raft_consensus.cc:705] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.124135 4804 consensus_queue.cc:229] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.132856 4811 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.133793 4804 ts_tablet_manager.cc:1087] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s I0720 00:43:58.134977 4805 ts_tablet_manager.cc:1087] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s I0720 00:43:58.135373 4805 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.135473 4810 tablet_bootstrap.cc:590] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.136718 4810 log.cc:525] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8: Log is configured to *not* fsync() on all Append() calls I0720 00:43:58.137092 4805 tablet_bootstrap.cc:590] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.137281 4809 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.137285 4813 tablet_bootstrap.cc:590] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.138310 4814 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.138330 4809 tablet_bootstrap.cc:590] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.135874 4811 tablet_bootstrap.cc:590] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.139339 4804 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.139719 4814 tablet_bootstrap.cc:590] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.138613 4812 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.140643 4804 tablet_bootstrap.cc:590] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.141567 4812 tablet_bootstrap.cc:590] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.142119 4813 log.cc:525] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Log is configured to *not* fsync() on all Append() calls I0720 00:43:58.142171 4805 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.142349 4805 ts_tablet_manager.cc:1059] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s I0720 00:43:58.143019 4805 raft_consensus.cc:340] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.143239 4809 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.143288 4805 raft_consensus.cc:366] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.143355 4805 raft_consensus.cc:705] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.143612 4809 ts_tablet_manager.cc:1059] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s I0720 00:43:58.143580 4805 consensus_queue.cc:229] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.143990 4814 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.144238 4814 ts_tablet_manager.cc:1059] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s I0720 00:43:58.144398 4813 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.144745 4813 ts_tablet_manager.cc:1059] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s I0720 00:43:58.145928 4811 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.146124 4811 ts_tablet_manager.cc:1059] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s I0720 00:43:58.146664 4805 ts_tablet_manager.cc:1087] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0720 00:43:58.146740 4812 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.146876 4812 ts_tablet_manager.cc:1059] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s I0720 00:43:58.146991 4805 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.146953 4803 raft_consensus.cc:340] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.147583 4803 raft_consensus.cc:366] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.148247 4805 tablet_bootstrap.cc:590] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.148330 4803 raft_consensus.cc:705] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.148797 4810 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.148705 4803 consensus_queue.cc:229] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.148985 4810 ts_tablet_manager.cc:1059] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.005s I0720 00:43:58.150422 4805 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.150605 4805 ts_tablet_manager.cc:1059] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I0720 00:43:58.151079 4803 ts_tablet_manager.cc:1087] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.039s user 0.000s sys 0.007s I0720 00:43:58.151314 4803 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.153340 4805 raft_consensus.cc:340] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.153632 4805 raft_consensus.cc:366] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.153671 4803 tablet_bootstrap.cc:590] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.153708 4805 raft_consensus.cc:705] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.154208 4805 consensus_queue.cc:229] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.154887 4804 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.155452 4805 ts_tablet_manager.cc:1087] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s I0720 00:43:58.156283 4804 ts_tablet_manager.cc:1059] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s I0720 00:43:58.156865 4814 raft_consensus.cc:340] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.157301 4804 raft_consensus.cc:340] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.157475 4814 raft_consensus.cc:366] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.157495 4804 raft_consensus.cc:366] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.157526 4814 raft_consensus.cc:705] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.157549 4804 raft_consensus.cc:705] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.157692 4804 consensus_queue.cc:229] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.157897 4814 consensus_queue.cc:229] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.157317 4813 raft_consensus.cc:340] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.156474 4805 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.158169 4813 raft_consensus.cc:366] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.158206 4813 raft_consensus.cc:705] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.155635 4803 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.158334 4813 consensus_queue.cc:229] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.158689 4812 raft_consensus.cc:340] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.158829 4812 raft_consensus.cc:366] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.158862 4812 raft_consensus.cc:705] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.158998 4812 consensus_queue.cc:229] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.159605 4530 heartbeater.cc:446] Master 127.0.58.190:42541 was elected leader, sending a full tablet report... I0720 00:43:58.160167 4813 ts_tablet_manager.cc:1087] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s I0720 00:43:58.160434 4813 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.161440 4813 tablet_bootstrap.cc:590] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.162186 4812 ts_tablet_manager.cc:1087] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.015s user 0.001s sys 0.003s I0720 00:43:58.162356 4812 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.163229 4812 tablet_bootstrap.cc:590] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.164181 4803 ts_tablet_manager.cc:1059] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s I0720 00:43:58.164479 4805 tablet_bootstrap.cc:590] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.165112 4814 ts_tablet_manager.cc:1087] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s I0720 00:43:58.165364 4814 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.165246 4803 raft_consensus.cc:340] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.165582 4803 raft_consensus.cc:366] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.165645 4803 raft_consensus.cc:705] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.165854 4803 consensus_queue.cc:229] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.166708 4813 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.167073 4813 ts_tablet_manager.cc:1059] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s I0720 00:43:58.167552 4813 raft_consensus.cc:340] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.167765 4813 raft_consensus.cc:366] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.167834 4813 raft_consensus.cc:705] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.166985 4814 tablet_bootstrap.cc:590] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.168931 4812 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.169064 4812 ts_tablet_manager.cc:1059] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s I0720 00:43:58.169185 4803 ts_tablet_manager.cc:1087] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s I0720 00:43:58.169051 4809 raft_consensus.cc:340] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.169478 4803 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.169518 4812 raft_consensus.cc:340] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.169878 4812 raft_consensus.cc:366] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.169930 4812 raft_consensus.cc:705] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.170219 4809 raft_consensus.cc:366] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.170163 4812 consensus_queue.cc:229] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.170013 4811 raft_consensus.cc:340] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.170385 4811 raft_consensus.cc:366] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.170433 4812 ts_tablet_manager.cc:1087] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0720 00:43:58.170485 4811 raft_consensus.cc:705] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.170630 4803 tablet_bootstrap.cc:590] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.170703 4812 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.171066 4811 consensus_queue.cc:229] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.169766 4810 raft_consensus.cc:340] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.171635 4810 raft_consensus.cc:366] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.171723 4810 raft_consensus.cc:705] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.171823 4805 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.171124 4813 consensus_queue.cc:229] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.171995 4805 ts_tablet_manager.cc:1059] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s I0720 00:43:58.171958 4810 consensus_queue.cc:229] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.172119 4813 ts_tablet_manager.cc:1087] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s I0720 00:43:58.172343 4813 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.172515 4805 raft_consensus.cc:340] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.172209 4812 tablet_bootstrap.cc:590] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.172729 4805 raft_consensus.cc:366] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.172760 4805 raft_consensus.cc:705] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.172911 4805 consensus_queue.cc:229] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.174178 4813 tablet_bootstrap.cc:590] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.174878 4805 ts_tablet_manager.cc:1087] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s I0720 00:43:58.175158 4805 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f: Bootstrap starting. I0720 00:43:58.174731 4814 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.175410 4814 ts_tablet_manager.cc:1059] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s I0720 00:43:58.175493 4809 raft_consensus.cc:705] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.175740 4809 consensus_queue.cc:229] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.175853 4814 raft_consensus.cc:340] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.176076 4814 raft_consensus.cc:366] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.176128 4814 raft_consensus.cc:705] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.176285 4805 tablet_bootstrap.cc:590] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f: No blocks or log segments found. Creating new log. I0720 00:43:58.176299 4814 consensus_queue.cc:229] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.176709 4655 heartbeater.cc:446] Master 127.0.58.190:42541 was elected leader, sending a full tablet report... I0720 00:43:58.175213 4804 ts_tablet_manager.cc:1087] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0720 00:43:58.177502 4803 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.177585 4803 ts_tablet_manager.cc:1059] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s I0720 00:43:58.177662 4810 ts_tablet_manager.cc:1087] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.029s user 0.003s sys 0.003s I0720 00:43:58.178014 4810 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.178025 4803 raft_consensus.cc:340] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.178179 4812 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.178278 4812 ts_tablet_manager.cc:1059] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s I0720 00:43:58.178278 4803 raft_consensus.cc:366] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.178324 4803 raft_consensus.cc:705] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.178333 4809 ts_tablet_manager.cc:1087] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s I0720 00:43:58.178516 4809 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.178534 4803 consensus_queue.cc:229] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.178907 4812 raft_consensus.cc:340] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.179170 4812 raft_consensus.cc:366] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.179253 4812 raft_consensus.cc:705] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.179543 4814 ts_tablet_manager.cc:1087] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0720 00:43:58.179539 4812 consensus_queue.cc:229] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.179793 4814 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.179920 4803 ts_tablet_manager.cc:1087] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:43:58.180217 4812 ts_tablet_manager.cc:1087] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:43:58.180554 4805 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f: No bootstrap required, opened a new log I0720 00:43:58.180428 4811 ts_tablet_manager.cc:1087] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.034s user 0.004s sys 0.004s I0720 00:43:58.180698 4805 ts_tablet_manager.cc:1059] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s I0720 00:43:58.180938 4811 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.181044 4812 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:43:58.181165 4805 raft_consensus.cc:340] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.181356 4805 raft_consensus.cc:366] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.181438 4805 raft_consensus.cc:705] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc86e9a330584f00b998288875c0586f, State: Initialized, Role: FOLLOWER I0720 00:43:58.181623 4805 consensus_queue.cc:229] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.181964 4812 tablet_bootstrap.cc:590] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.182036 4805 ts_tablet_manager.cc:1087] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0720 00:43:58.180879 4814 tablet_bootstrap.cc:590] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: No blocks or log segments found. Creating new log. I0720 00:43:58.184608 4813 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.184783 4813 ts_tablet_manager.cc:1059] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s I0720 00:43:58.185438 4813 raft_consensus.cc:340] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.185673 4813 raft_consensus.cc:366] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.185739 4813 raft_consensus.cc:705] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.185851 4811 tablet_bootstrap.cc:590] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.186921 4810 tablet_bootstrap.cc:590] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.187014 4809 tablet_bootstrap.cc:590] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.185962 4813 consensus_queue.cc:229] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.188848 4811 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.189020 4811 ts_tablet_manager.cc:1059] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s I0720 00:43:58.189198 4813 ts_tablet_manager.cc:1087] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s I0720 00:43:58.189597 4811 raft_consensus.cc:340] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.189855 4811 raft_consensus.cc:366] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.189916 4811 raft_consensus.cc:705] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.190192 4811 consensus_queue.cc:229] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.192363 4814 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.192519 4814 ts_tablet_manager.cc:1059] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s I0720 00:43:58.193101 4814 raft_consensus.cc:340] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.193375 4814 raft_consensus.cc:366] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.193415 4811 ts_tablet_manager.cc:1087] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0720 00:43:58.193462 4814 raft_consensus.cc:705] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.193670 4811 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.193670 4814 consensus_queue.cc:229] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.194583 4814 ts_tablet_manager.cc:1087] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:43:58.194895 4812 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: No bootstrap required, opened a new log I0720 00:43:58.195448 4812 ts_tablet_manager.cc:1059] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s I0720 00:43:58.195909 4809 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.196072 4809 ts_tablet_manager.cc:1059] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s I0720 00:43:58.196727 4809 raft_consensus.cc:340] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.196907 4809 raft_consensus.cc:366] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.196841 4812 raft_consensus.cc:340] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.196965 4809 raft_consensus.cc:705] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.197180 4812 raft_consensus.cc:366] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.197176 4809 consensus_queue.cc:229] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.197268 4812 raft_consensus.cc:705] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506b8430b3284890b468e7ef55b6ca79, State: Initialized, Role: FOLLOWER I0720 00:43:58.197264 4811 tablet_bootstrap.cc:590] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.197461 4812 consensus_queue.cc:229] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.198441 4812 ts_tablet_manager.cc:1087] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:43:58.198781 4809 ts_tablet_manager.cc:1087] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s I0720 00:43:58.199036 4809 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.199101 4810 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.199254 4810 ts_tablet_manager.cc:1059] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s I0720 00:43:58.199997 4810 raft_consensus.cc:340] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.200152 4810 raft_consensus.cc:366] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.200232 4809 tablet_bootstrap.cc:590] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.200237 4810 raft_consensus.cc:705] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.200610 4810 consensus_queue.cc:229] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.201539 4810 ts_tablet_manager.cc:1087] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:43:58.201669 4811 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.201869 4810 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.201925 4811 ts_tablet_manager.cc:1059] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s I0720 00:43:58.202675 4811 raft_consensus.cc:340] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.202939 4811 raft_consensus.cc:366] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.203058 4811 raft_consensus.cc:705] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.203328 4811 consensus_queue.cc:229] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.203598 4810 tablet_bootstrap.cc:590] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.203866 4809 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.204021 4809 ts_tablet_manager.cc:1059] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s I0720 00:43:58.204175 4811 ts_tablet_manager.cc:1087] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:43:58.204447 4811 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8: Bootstrap starting. I0720 00:43:58.204581 4809 raft_consensus.cc:340] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } W0720 00:43:58.204717 4780 tablet.cc:1799] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:43:58.204828 4809 raft_consensus.cc:366] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.204874 4809 raft_consensus.cc:705] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.205032 4809 consensus_queue.cc:229] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.205415 4809 ts_tablet_manager.cc:1087] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0720 00:43:58.205670 4811 tablet_bootstrap.cc:590] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8: No blocks or log segments found. Creating new log. I0720 00:43:58.207031 4810 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.207167 4810 ts_tablet_manager.cc:1059] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s I0720 00:43:58.207396 4811 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8: No bootstrap required, opened a new log I0720 00:43:58.207535 4811 ts_tablet_manager.cc:1059] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s I0720 00:43:58.207614 4810 raft_consensus.cc:340] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.207824 4810 raft_consensus.cc:366] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.207866 4810 raft_consensus.cc:705] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.208006 4810 consensus_queue.cc:229] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.208215 4811 raft_consensus.cc:340] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.208364 4810 ts_tablet_manager.cc:1087] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0720 00:43:58.208405 4811 raft_consensus.cc:366] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:43:58.208456 4811 raft_consensus.cc:705] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Initialized, Role: FOLLOWER I0720 00:43:58.208592 4811 consensus_queue.cc:229] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.208932 4811 ts_tablet_manager.cc:1087] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s I0720 00:43:58.210044 4837 raft_consensus.cc:472] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.210260 4837 raft_consensus.cc:494] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.211477 4837 leader_election.cc:296] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.221773 4815 raft_consensus.cc:472] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.221993 4815 raft_consensus.cc:494] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.223271 4815 leader_election.cc:296] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:58.231766 4815 raft_consensus.cc:472] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.231950 4815 raft_consensus.cc:494] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.232337 4837 raft_consensus.cc:472] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.232376 4815 leader_election.cc:296] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) I0720 00:43:58.232440 4837 raft_consensus.cc:494] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.232682 4815 raft_consensus.cc:472] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.232774 4815 raft_consensus.cc:494] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.232875 4837 leader_election.cc:296] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.233392 4815 leader_election.cc:296] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) I0720 00:43:58.233817 4487 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca6a2a19cbbe4648a8a76de1e43c405b" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.234347 4487 raft_consensus.cc:2350] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.235064 4487 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27285c1e3c36466f9bce8450be7500d4" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.235205 4487 raft_consensus.cc:2350] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.235718 4544 leader_election.cc:310] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.236178 4837 raft_consensus.cc:2644] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.236284 4837 raft_consensus.cc:472] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.236384 4837 raft_consensus.cc:2890] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.236788 4487 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ac7512c8584fe3ad887ffef570df8d" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.236953 4487 raft_consensus.cc:2350] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.237421 4826 raft_consensus.cc:472] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.237565 4826 raft_consensus.cc:494] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.238358 4837 raft_consensus.cc:494] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.238725 4826 leader_election.cc:296] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.238905 4837 leader_election.cc:296] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.239207 4544 leader_election.cc:310] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.239329 4487 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5758df4920ce4bbab0424048f13fea41" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.239451 4837 raft_consensus.cc:2644] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.239464 4487 raft_consensus.cc:2350] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.239506 4837 raft_consensus.cc:472] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.239542 4837 raft_consensus.cc:2890] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.241066 4837 raft_consensus.cc:494] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.241541 4837 leader_election.cc:296] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.241883 4487 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca6a2a19cbbe4648a8a76de1e43c405b" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.242025 4487 raft_consensus.cc:2890] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.242544 4668 leader_election.cc:310] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.242657 4736 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca6a2a19cbbe4648a8a76de1e43c405b" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.243050 4736 raft_consensus.cc:2350] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.242172 4486 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "19920b309dd84a01b2cc74ddbbd10854" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.243394 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27285c1e3c36466f9bce8450be7500d4" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.243672 4668 leader_election.cc:310] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.243651 4485 raft_consensus.cc:2890] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.243906 4815 raft_consensus.cc:2644] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.243458 4486 raft_consensus.cc:2350] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.243976 4815 raft_consensus.cc:472] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.244021 4815 raft_consensus.cc:2890] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.245750 4487 raft_consensus.cc:2350] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.245864 4815 raft_consensus.cc:494] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.246402 4485 raft_consensus.cc:2350] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.246515 4815 leader_election.cc:296] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:58.246632 4815 raft_consensus.cc:2644] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.246675 4815 raft_consensus.cc:472] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.246724 4815 raft_consensus.cc:2890] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.248117 4815 raft_consensus.cc:494] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.248546 4815 leader_election.cc:296] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Requested vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) I0720 00:43:58.248837 4668 leader_election.cc:310] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.249081 4815 raft_consensus.cc:2644] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.249173 4815 raft_consensus.cc:472] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.249244 4815 raft_consensus.cc:2890] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.249357 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ac7512c8584fe3ad887ffef570df8d" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.249487 4485 raft_consensus.cc:2890] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.251008 4485 raft_consensus.cc:2350] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.251451 4668 leader_election.cc:310] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.251714 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5758df4920ce4bbab0424048f13fea41" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.251670 4815 raft_consensus.cc:494] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.251943 4485 raft_consensus.cc:2890] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.252146 4815 leader_election.cc:296] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Requested vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) I0720 00:43:58.252303 4815 raft_consensus.cc:2644] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.252424 4815 raft_consensus.cc:667] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Becoming Leader. State: Replica: fc86e9a330584f00b998288875c0586f, State: Running, Role: LEADER I0720 00:43:58.253567 4485 raft_consensus.cc:2350] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.254182 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "19920b309dd84a01b2cc74ddbbd10854" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.254351 4485 raft_consensus.cc:2890] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.255707 4485 raft_consensus.cc:2350] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.245312 4834 raft_consensus.cc:472] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.256429 4834 raft_consensus.cc:494] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.256955 4834 leader_election.cc:296] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.246873 4544 leader_election.cc:310] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.257442 4544 leader_election.cc:310] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.257468 4668 leader_election.cc:310] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.257591 4837 raft_consensus.cc:2644] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.257941 4834 raft_consensus.cc:2644] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.258020 4834 raft_consensus.cc:667] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Becoming Leader. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Running, Role: LEADER I0720 00:43:58.258653 4834 consensus_queue.cc:206] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.263408 4837 raft_consensus.cc:667] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Becoming Leader. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Running, Role: LEADER I0720 00:43:58.263715 4837 consensus_queue.cc:206] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.263876 4815 consensus_queue.cc:206] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.267053 4826 raft_consensus.cc:472] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.267328 4826 raft_consensus.cc:494] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.267345 4612 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ac7512c8584fe3ad887ffef570df8d" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.267778 4826 leader_election.cc:296] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.267972 4611 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5758df4920ce4bbab0424048f13fea41" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.266798 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9ef65edd881743f691ebb903dad80c36" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.268563 4485 raft_consensus.cc:2350] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.269291 4857 raft_consensus.cc:2644] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.267711 4612 raft_consensus.cc:2350] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.269510 4852 raft_consensus.cc:472] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.268187 4611 raft_consensus.cc:2350] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.270575 4610 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "19920b309dd84a01b2cc74ddbbd10854" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.271127 4855 raft_consensus.cc:472] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.271036 4610 raft_consensus.cc:2350] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.271733 4544 leader_election.cc:310] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.271781 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ac7512c8584fe3ad887ffef570df8d" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" I0720 00:43:58.271994 4609 raft_consensus.cc:2890] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.271509 4668 leader_election.cc:310] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.273777 4609 raft_consensus.cc:2350] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.272965 4362 catalog_manager.cc:4067] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: term changed from 0 to 1, leader changed from to a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: UNKNOWN } } } I0720 00:43:58.270933 4857 raft_consensus.cc:667] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Becoming Leader. State: Replica: fc86e9a330584f00b998288875c0586f, State: Running, Role: LEADER I0720 00:43:58.271473 4852 raft_consensus.cc:494] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.276343 4736 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27285c1e3c36466f9bce8450be7500d4" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.276490 4736 raft_consensus.cc:2350] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.271289 4855 raft_consensus.cc:494] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.277307 4834 raft_consensus.cc:2644] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.277411 4834 raft_consensus.cc:472] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.277014 4735 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "ca6a2a19cbbe4648a8a76de1e43c405b" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" I0720 00:43:58.277514 4834 raft_consensus.cc:2890] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.277631 4735 raft_consensus.cc:2890] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.278164 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7869d31b2ece4c38b8cf73ba177d2787" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.278092 4734 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27285c1e3c36466f9bce8450be7500d4" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" I0720 00:43:58.278326 4485 raft_consensus.cc:2350] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.278371 4734 raft_consensus.cc:2890] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.278043 4733 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9ef65edd881743f691ebb903dad80c36" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.277323 4854 raft_consensus.cc:2644] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.278683 4733 raft_consensus.cc:2350] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.278812 4736 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5758df4920ce4bbab0424048f13fea41" candidate_uuid: "506b8430b3284890b468e7ef55b6ca79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.280382 4833 raft_consensus.cc:472] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.280581 4833 raft_consensus.cc:494] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.281549 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c4890cfc794b469d8765701546219a19" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.281697 4485 raft_consensus.cc:2350] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.282205 4544 leader_election.cc:310] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.280566 4732 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9ef65edd881743f691ebb903dad80c36" candidate_uuid: "506b8430b3284890b468e7ef55b6ca79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.282182 4833 leader_election.cc:296] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.282582 4833 raft_consensus.cc:2644] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.280311 4834 raft_consensus.cc:494] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.282898 4833 raft_consensus.cc:472] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.282405 4732 raft_consensus.cc:2350] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506b8430b3284890b468e7ef55b6ca79 in term 0. I0720 00:43:58.282972 4833 raft_consensus.cc:2890] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.281673 4852 leader_election.cc:296] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:58.280731 4668 leader_election.cc:310] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.283646 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5758df4920ce4bbab0424048f13fea41" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" I0720 00:43:58.283839 4852 raft_consensus.cc:2644] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.284253 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9ef65edd881743f691ebb903dad80c36" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.284389 4485 raft_consensus.cc:2890] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.285848 4485 raft_consensus.cc:2350] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.286381 4609 raft_consensus.cc:2890] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.287075 4734 raft_consensus.cc:2350] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.287194 4852 raft_consensus.cc:472] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.289885 4609 raft_consensus.cc:2350] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.290123 4833 raft_consensus.cc:494] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.290518 4833 leader_election.cc:296] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.290735 4544 leader_election.cc:310] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.290959 4834 leader_election.cc:296] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.291468 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c4890cfc794b469d8765701546219a19" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.291617 4485 raft_consensus.cc:2890] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.283363 4857 consensus_queue.cc:206] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.292968 4485 raft_consensus.cc:2350] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.291082 4833 raft_consensus.cc:2644] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.293684 4833 raft_consensus.cc:667] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Becoming Leader. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Running, Role: LEADER I0720 00:43:58.293884 4833 consensus_queue.cc:206] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.294956 4544 leader_election.cc:310] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.295076 4855 leader_election.cc:296] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527), 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) I0720 00:43:58.296321 4833 raft_consensus.cc:2644] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.296561 4833 raft_consensus.cc:667] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Becoming Leader. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Running, Role: LEADER I0720 00:43:58.296743 4833 consensus_queue.cc:206] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.299073 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "19920b309dd84a01b2cc74ddbbd10854" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" I0720 00:43:58.299190 4609 raft_consensus.cc:2890] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.299255 4418 leader_election.cc:310] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.299454 4362 catalog_manager.cc:4067] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: term changed from 0 to 1, leader changed from to a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: UNKNOWN } } } I0720 00:43:58.299837 4826 raft_consensus.cc:2607] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won I0720 00:43:58.299952 4855 raft_consensus.cc:472] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.300074 4855 raft_consensus.cc:494] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.300498 4852 raft_consensus.cc:2890] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.300503 4609 raft_consensus.cc:2350] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.300678 4855 leader_election.cc:296] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:58.301087 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c4890cfc794b469d8765701546219a19" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.301287 4485 raft_consensus.cc:2280] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fc86e9a330584f00b998288875c0586f in current term 1: Already voted for candidate a29a463e4c2e4f61a400dde426c625e8 in this term. I0720 00:43:58.301483 4734 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c4890cfc794b469d8765701546219a19" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.302429 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7869d31b2ece4c38b8cf73ba177d2787" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.302461 4364 catalog_manager.cc:4067] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f reported cstate change: term changed from 0 to 1, leader changed from to fc86e9a330584f00b998288875c0586f (127.0.58.131). New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:43:58.303241 4609 raft_consensus.cc:2350] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.303525 4734 raft_consensus.cc:2350] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.303735 4732 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c4890cfc794b469d8765701546219a19" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" I0720 00:43:58.303834 4732 raft_consensus.cc:2890] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.304111 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5758df4920ce4bbab0424048f13fea41" candidate_uuid: "506b8430b3284890b468e7ef55b6ca79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.304284 4609 raft_consensus.cc:2280] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506b8430b3284890b468e7ef55b6ca79 in current term 1: Already voted for candidate fc86e9a330584f00b998288875c0586f in this term. I0720 00:43:58.304302 4362 catalog_manager.cc:4067] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: term changed from 0 to 1, leader changed from to a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: UNKNOWN } } } I0720 00:43:58.304615 4362 catalog_manager.cc:4067] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: term changed from 0 to 1, leader changed from to a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: UNKNOWN } } } I0720 00:43:58.304828 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c4890cfc794b469d8765701546219a19" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.304989 4609 raft_consensus.cc:2311] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate fc86e9a330584f00b998288875c0586f for term 1 because replica is either leader or believes a valid leader to be alive. I0720 00:43:58.305303 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9ef65edd881743f691ebb903dad80c36" candidate_uuid: "506b8430b3284890b468e7ef55b6ca79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.305435 4609 raft_consensus.cc:2311] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 506b8430b3284890b468e7ef55b6ca79 for term 1 because replica is either leader or believes a valid leader to be alive. I0720 00:43:58.305442 4733 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9ef65edd881743f691ebb903dad80c36" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" I0720 00:43:58.305549 4733 raft_consensus.cc:2890] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.305744 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0a7ad36f80f44067ba0087a2768e26ac" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" is_pre_election: true I0720 00:43:58.305882 4609 raft_consensus.cc:2350] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.307305 4667 leader_election.cc:310] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29a463e4c2e4f61a400dde426c625e8, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.307544 4668 leader_election.cc:310] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc86e9a330584f00b998288875c0586f; no voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8 I0720 00:43:58.308037 4735 raft_consensus.cc:2350] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.308679 4861 raft_consensus.cc:2644] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.308807 4732 raft_consensus.cc:2350] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.309610 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0a7ad36f80f44067ba0087a2768e26ac" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.309782 4485 raft_consensus.cc:2350] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 0. I0720 00:43:58.307626 4854 raft_consensus.cc:667] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Becoming Leader. State: Replica: fc86e9a330584f00b998288875c0586f, State: Running, Role: LEADER I0720 00:43:58.310180 4733 raft_consensus.cc:2350] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.308828 4861 raft_consensus.cc:472] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.310483 4855 raft_consensus.cc:2592] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0720 00:43:58.310879 4861 raft_consensus.cc:2890] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.310953 4736 raft_consensus.cc:2311] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 506b8430b3284890b468e7ef55b6ca79 for term 1 because replica is either leader or believes a valid leader to be alive. I0720 00:43:58.311121 4852 raft_consensus.cc:494] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.312031 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7869d31b2ece4c38b8cf73ba177d2787" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" I0720 00:43:58.312183 4609 raft_consensus.cc:2890] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.311408 4854 consensus_queue.cc:206] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.313756 4609 raft_consensus.cc:2350] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.313622 4861 raft_consensus.cc:494] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.314100 4667 leader_election.cc:310] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29a463e4c2e4f61a400dde426c625e8, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.314232 4861 leader_election.cc:296] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:58.314378 4861 raft_consensus.cc:2644] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.314534 4861 raft_consensus.cc:667] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Becoming Leader. State: Replica: fc86e9a330584f00b998288875c0586f, State: Running, Role: LEADER I0720 00:43:58.314702 4609 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0a7ad36f80f44067ba0087a2768e26ac" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a29a463e4c2e4f61a400dde426c625e8" I0720 00:43:58.314836 4609 raft_consensus.cc:2890] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.314785 4861 consensus_queue.cc:206] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.314847 4362 catalog_manager.cc:4067] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f reported cstate change: term changed from 0 to 1, leader changed from to fc86e9a330584f00b998288875c0586f (127.0.58.131). New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:43:58.315165 4362 catalog_manager.cc:4067] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f reported cstate change: term changed from 0 to 1, leader changed from to fc86e9a330584f00b998288875c0586f (127.0.58.131). New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:43:58.316138 4609 raft_consensus.cc:2350] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.316467 4667 leader_election.cc:310] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29a463e4c2e4f61a400dde426c625e8, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.316478 4418 leader_election.cc:310] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79; no voters: a29a463e4c2e4f61a400dde426c625e8, fc86e9a330584f00b998288875c0586f I0720 00:43:58.316741 4854 raft_consensus.cc:2644] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.316810 4826 raft_consensus.cc:2592] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0720 00:43:58.316867 4854 raft_consensus.cc:667] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Becoming Leader. State: Replica: fc86e9a330584f00b998288875c0586f, State: Running, Role: LEADER I0720 00:43:58.317028 4854 consensus_queue.cc:206] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.317306 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7869d31b2ece4c38b8cf73ba177d2787" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.317447 4485 raft_consensus.cc:2890] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.318922 4485 raft_consensus.cc:2350] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. I0720 00:43:58.319115 4852 leader_election.cc:296] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527) I0720 00:43:58.319545 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0a7ad36f80f44067ba0087a2768e26ac" candidate_uuid: "fc86e9a330584f00b998288875c0586f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.319772 4485 raft_consensus.cc:2890] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.320773 4362 catalog_manager.cc:4067] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f reported cstate change: term changed from 0 to 1, leader changed from to fc86e9a330584f00b998288875c0586f (127.0.58.131). New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:43:58.321074 4362 catalog_manager.cc:4067] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f reported cstate change: term changed from 0 to 1, leader changed from to fc86e9a330584f00b998288875c0586f (127.0.58.131). New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:43:58.321318 4485 raft_consensus.cc:2350] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc86e9a330584f00b998288875c0586f in term 1. W0720 00:43:58.337352 4531 tablet.cc:1799] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Can't schedule compaction. Clean time has not been advanced past its initial value. W0720 00:43:58.375171 4656 tablet.cc:1799] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:43:58.421604 4837 raft_consensus.cc:472] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:43:58.421864 4837 raft_consensus.cc:494] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.422638 4837 leader_election.cc:296] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.423466 4736 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ea05088cb24563bcf684d6199dd946" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" is_pre_election: true I0720 00:43:58.423466 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ea05088cb24563bcf684d6199dd946" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" is_pre_election: true I0720 00:43:58.423674 4736 raft_consensus.cc:2350] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.423676 4485 raft_consensus.cc:2350] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 0. I0720 00:43:58.424116 4543 leader_election.cc:310] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a29a463e4c2e4f61a400dde426c625e8, fc86e9a330584f00b998288875c0586f; no voters: I0720 00:43:58.424466 4837 raft_consensus.cc:2644] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:43:58.424528 4837 raft_consensus.cc:472] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:43:58.424592 4837 raft_consensus.cc:2890] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.426442 4837 raft_consensus.cc:494] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.426915 4837 leader_election.cc:296] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Requested vote from peers 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375), fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) I0720 00:43:58.427419 4485 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ea05088cb24563bcf684d6199dd946" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506b8430b3284890b468e7ef55b6ca79" I0720 00:43:58.427475 4736 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "73ea05088cb24563bcf684d6199dd946" candidate_uuid: "a29a463e4c2e4f61a400dde426c625e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc86e9a330584f00b998288875c0586f" I0720 00:43:58.427559 4485 raft_consensus.cc:2890] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.427621 4736 raft_consensus.cc:2890] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:43:58.429221 4485 raft_consensus.cc:2350] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.429332 4736 raft_consensus.cc:2350] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a29a463e4c2e4f61a400dde426c625e8 in term 1. I0720 00:43:58.429675 4544 leader_election.cc:310] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506b8430b3284890b468e7ef55b6ca79, a29a463e4c2e4f61a400dde426c625e8; no voters: I0720 00:43:58.430006 4837 raft_consensus.cc:2644] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:43:58.430171 4837 raft_consensus.cc:667] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Becoming Leader. State: Replica: a29a463e4c2e4f61a400dde426c625e8, State: Running, Role: LEADER I0720 00:43:58.430414 4837 consensus_queue.cc:206] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:43:58.432948 4362 catalog_manager.cc:4067] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: term changed from 0 to 1, leader changed from to a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: UNKNOWN } } } I0720 00:43:58.534605 4609 raft_consensus.cc:1184] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.536497 4485 raft_consensus.cc:1184] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.537613 4854 consensus_queue.cc:984] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:58.548422 4852 consensus_queue.cc:984] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s I0720 00:43:58.561231 4610 raft_consensus.cc:1184] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.563033 4610 raft_consensus.cc:1184] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.564950 4487 raft_consensus.cc:1184] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.565927 4861 consensus_queue.cc:984] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s I0720 00:43:58.567231 4487 raft_consensus.cc:1184] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.568266 4854 consensus_queue.cc:984] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s I0720 00:43:58.569505 4855 consensus_queue.cc:984] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s I0720 00:43:58.570192 4487 raft_consensus.cc:1184] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.571045 4487 raft_consensus.cc:1184] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.572361 4736 raft_consensus.cc:1184] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.572753 4857 consensus_queue.cc:984] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:43:58.575896 4852 consensus_queue.cc:984] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s I0720 00:43:58.571688 4833 consensus_queue.cc:984] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s I0720 00:43:58.611424 4833 consensus_queue.cc:984] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s I0720 00:43:58.613862 4485 raft_consensus.cc:1184] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.614643 4612 raft_consensus.cc:1184] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.621368 4484 raft_consensus.cc:1184] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.622740 4484 raft_consensus.cc:1184] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.623464 4854 consensus_queue.cc:984] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s I0720 00:43:58.624583 4735 raft_consensus.cc:1184] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.624939 4833 consensus_queue.cc:984] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:43:58.625725 4732 raft_consensus.cc:1184] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.625949 4736 raft_consensus.cc:1184] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.627647 4837 consensus_queue.cc:984] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s I0720 00:43:58.628284 4837 consensus_queue.cc:984] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:58.629248 4891 mvcc.cc:203] Tried to move safe_time back from 6404437764375064576 to 6404437763254026240. Current Snapshot: MvccSnapshot[committed={T|T < 6404437764375064576 or (T in {6404437764375064576})}] I0720 00:43:58.649251 4878 mvcc.cc:203] Tried to move safe_time back from 6404437764350472192 to 6404437763339227136. Current Snapshot: MvccSnapshot[committed={T|T < 6404437764350472192 or (T in {6404437764350472192})}] I0720 00:43:58.649627 4833 consensus_queue.cc:984] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s I0720 00:43:58.662036 4480 raft_consensus.cc:1184] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.663836 4815 consensus_queue.cc:984] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:58.678948 4733 raft_consensus.cc:1184] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:43:58.679286 4608 raft_consensus.cc:1184] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:43:58.679481 4837 consensus_queue.cc:984] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s I0720 00:43:58.680182 4837 consensus_queue.cc:984] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:58.694015 4886 mvcc.cc:203] Tried to move safe_time back from 6404437764375064576 to 6404437763254026240. Current Snapshot: MvccSnapshot[committed={T|T < 6404437764375064576}] I0720 00:43:58.701791 4479 raft_consensus.cc:1184] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:43:58.703188 4857 consensus_queue.cc:984] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:58.705346 4834 consensus_queue.cc:984] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506b8430b3284890b468e7ef55b6ca79" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:43:58.741117 4906 consensus_queue.cc:984] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s W0720 00:43:58.900738 4905 log.cc:669] Time spent T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8: Append to log took a long time: real 0.054s user 0.000s sys 0.000s W0720 00:43:58.963032 4544 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:36375, user_credentials={real_user=slave}} blocked reactor thread for 49516.3us I0720 00:43:59.206418 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000286 I0720 00:43:59.208680 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000299 I0720 00:43:59.276273 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.069s user 0.024s sys 0.001s Metrics: {"bytes_written":7323,"cfile_init":1,"compiler_manager_pool.queue_time_us":4541,"data dirs.queue_time_us":722,"data dirs.run_cpu_time_us":146,"data dirs.run_wall_time_us":6250,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":545,"lbm_writes_lt_1ms":23,"rows_written":103,"thread_start_us":4763,"threads_started":2} I0720 00:43:59.277803 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000305 I0720 00:43:59.289791 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.081s user 0.022s sys 0.004s Metrics: {"bytes_written":7024,"cfile_init":1,"compiler_manager_pool.queue_time_us":5730,"data dirs.queue_time_us":2697,"data dirs.run_cpu_time_us":163,"data dirs.run_wall_time_us":2353,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"rows_written":90,"thread_start_us":5055,"threads_started":2} I0720 00:43:59.291362 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 967 bytes on disk I0720 00:43:59.292132 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:43:59.292780 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000322 I0720 00:43:59.303972 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6854,"cfile_init":1,"data dirs.queue_time_us":56,"data dirs.run_cpu_time_us":149,"data dirs.run_wall_time_us":1541,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"rows_written":84} I0720 00:43:59.304935 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 931 bytes on disk I0720 00:43:59.305428 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:43:59.305948 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000322 I0720 00:43:59.320657 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.014s user 0.007s sys 0.003s Metrics: {"bytes_written":6895,"cfile_init":1,"data dirs.queue_time_us":2430,"data dirs.run_cpu_time_us":147,"data dirs.run_wall_time_us":1266,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":453,"lbm_writes_lt_1ms":23,"rows_written":86,"thread_start_us":2269,"threads_started":1} I0720 00:43:59.322698 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 958 bytes on disk I0720 00:43:59.323151 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:43:59.325398 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000326 I0720 00:43:59.341542 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000328 I0720 00:43:59.354982 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.029s user 0.014s sys 0.000s Metrics: {"bytes_written":7826,"cfile_init":1,"data dirs.queue_time_us":3030,"data dirs.run_cpu_time_us":162,"data dirs.run_wall_time_us":1237,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":467,"lbm_writes_lt_1ms":23,"rows_written":121,"thread_start_us":2840,"threads_started":1} I0720 00:43:59.356040 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000333 I0720 00:43:59.371217 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.015s user 0.007s sys 0.004s Metrics: {"bytes_written":7528,"cfile_init":1,"data dirs.queue_time_us":90,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":1506,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"rows_written":112} I0720 00:43:59.372184 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000337 I0720 00:43:59.387418 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.015s user 0.009s sys 0.001s Metrics: {"bytes_written":7103,"cfile_init":1,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":112,"data dirs.run_wall_time_us":542,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":23,"rows_written":95} I0720 00:43:59.388325 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 1270 bytes on disk I0720 00:43:59.388813 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4} I0720 00:43:59.389416 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000339 I0720 00:43:59.402436 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.060s user 0.027s sys 0.000s Metrics: {"bytes_written":7289,"cfile_init":1,"compiler_manager_pool.queue_time_us":807,"data dirs.queue_time_us":2321,"data dirs.run_cpu_time_us":170,"data dirs.run_wall_time_us":965,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"rows_written":101,"thread_start_us":2815,"threads_started":2} I0720 00:43:59.404048 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 1102 bytes on disk I0720 00:43:59.404737 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4} I0720 00:43:59.405395 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000345 I0720 00:43:59.420217 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.030s user 0.009s sys 0.000s Metrics: {"bytes_written":7649,"cfile_init":1,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":463,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1180,"lbm_writes_lt_1ms":23,"rows_written":115} I0720 00:43:59.422830 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 1211 bytes on disk I0720 00:43:59.423316 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4} I0720 00:43:59.424520 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000347 I0720 00:43:59.426326 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.021s user 0.010s sys 0.006s Metrics: {"bytes_written":7442,"cfile_init":1,"data dirs.queue_time_us":3007,"data dirs.run_cpu_time_us":138,"data dirs.run_wall_time_us":1101,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2645,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":106,"thread_start_us":2814,"threads_started":1} I0720 00:43:59.427232 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 1127 bytes on disk I0720 00:43:59.427701 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:43:59.428234 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000351 I0720 00:43:59.442098 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.014s user 0.006s sys 0.004s Metrics: {"bytes_written":7649,"cfile_init":1,"data dirs.queue_time_us":66,"data dirs.run_cpu_time_us":119,"data dirs.run_wall_time_us":508,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"rows_written":115} I0720 00:43:59.452752 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000354 I0720 00:43:59.461155 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.182s user 0.005s sys 0.006s Metrics: {"bytes_written":6892,"cfile_init":1,"data dirs.queue_time_us":809,"data dirs.run_cpu_time_us":148,"data dirs.run_wall_time_us":1135,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":37873,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"rows_written":86,"thread_start_us":4981,"threads_started":1} I0720 00:43:59.462096 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000356 I0720 00:43:59.508256 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.046s user 0.007s sys 0.003s Metrics: {"bytes_written":7524,"cfile_init":1,"data dirs.queue_time_us":1067,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"rows_written":112} I0720 00:43:59.509124 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000365 I0720 00:43:59.535177 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.026s user 0.003s sys 0.008s Metrics: {"bytes_written":8068,"cfile_init":1,"data dirs.queue_time_us":2024,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"rows_written":130,"thread_start_us":5930,"threads_started":1} I0720 00:43:59.536062 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 1378 bytes on disk I0720 00:43:59.536752 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4} I0720 00:43:59.537318 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000372 I0720 00:43:59.539512 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.085s user 0.009s sys 0.002s Metrics: {"bytes_written":7488,"cfile_init":1,"data dirs.queue_time_us":3652,"data dirs.run_cpu_time_us":121,"data dirs.run_wall_time_us":8059,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"rows_written":108,"thread_start_us":6790,"threads_started":1} I0720 00:43:59.542757 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 1211 bytes on disk I0720 00:43:59.543267 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4} I0720 00:43:59.543834 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000381 I0720 00:43:59.599762 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.175s user 0.006s sys 0.004s Metrics: {"bytes_written":7665,"cfile_init":1,"data dirs.queue_time_us":4753,"data dirs.run_cpu_time_us":112,"data dirs.run_wall_time_us":631,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":491,"lbm_writes_lt_1ms":23,"rows_written":115,"spinlock_wait_cycles":15232} I0720 00:43:59.606840 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000396 I0720 00:43:59.636624 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.099s user 0.009s sys 0.002s Metrics: {"bytes_written":7944,"cfile_init":1,"data dirs.queue_time_us":39346,"data dirs.run_cpu_time_us":166,"data dirs.run_wall_time_us":865,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":499,"lbm_writes_lt_1ms":23,"rows_written":127} I0720 00:43:59.637482 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000399 I0720 00:43:59.642550 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.035s user 0.007s sys 0.006s Metrics: {"bytes_written":8360,"cfile_init":1,"data dirs.queue_time_us":2729,"data dirs.run_cpu_time_us":101,"data dirs.run_wall_time_us":528,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"rows_written":144,"spinlock_wait_cycles":20352} I0720 00:43:59.643615 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000406 I0720 00:43:59.655786 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.112s user 0.011s sys 0.000s Metrics: {"bytes_written":8129,"cfile_init":1,"data dirs.queue_time_us":25144,"data dirs.run_cpu_time_us":297,"data dirs.run_wall_time_us":867,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14055,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":134} I0720 00:43:59.656771 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 1426 bytes on disk I0720 00:43:59.657346 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:43:59.657939 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 1167 bytes on disk I0720 00:43:59.659448 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4} I0720 00:43:59.665047 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.021s user 0.008s sys 0.004s Metrics: {"bytes_written":8678,"cfile_init":1,"data dirs.queue_time_us":2289,"data dirs.run_cpu_time_us":139,"data dirs.run_wall_time_us":543,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":460,"lbm_writes_lt_1ms":23,"rows_written":154,"spinlock_wait_cycles":4352} I0720 00:43:59.666049 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 1621 bytes on disk I0720 00:43:59.666669 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4} I0720 00:43:59.667258 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 1518 bytes on disk I0720 00:43:59.667629 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000413 I0720 00:43:59.667698 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4} I0720 00:43:59.668279 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 1232 bytes on disk I0720 00:43:59.668663 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4} I0720 00:43:59.669167 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 1189 bytes on disk I0720 00:43:59.669613 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4} I0720 00:43:59.670162 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 1023 bytes on disk I0720 00:43:59.670559 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4} I0720 00:43:59.879093 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.241s user 0.014s sys 0.000s Metrics: {"bytes_written":8929,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":170,"data dirs.run_wall_time_us":898,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13241,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":165} I0720 00:43:59.880136 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000466 I0720 00:43:59.889379 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.222s user 0.013s sys 0.000s Metrics: {"bytes_written":8971,"cfile_init":1,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":141,"data dirs.run_wall_time_us":467,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":501,"lbm_writes_lt_1ms":23,"rows_written":167} I0720 00:43:59.890354 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 1727 bytes on disk I0720 00:43:59.890866 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:43:59.891398 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000475 I0720 00:43:59.985924 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.105s user 0.012s sys 0.002s Metrics: {"bytes_written":9323,"cfile_init":1,"data dirs.queue_time_us":3175,"data dirs.run_cpu_time_us":144,"data dirs.run_wall_time_us":992,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":780,"lbm_writes_lt_1ms":23,"rows_written":179,"spinlock_wait_cycles":13056} I0720 00:44:00.002421 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000499 I0720 00:44:00.034211 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.143s user 0.015s sys 0.000s Metrics: {"bytes_written":10391,"cfile_init":1,"data dirs.queue_time_us":5000,"data dirs.run_cpu_time_us":166,"data dirs.run_wall_time_us":1367,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"rows_written":222,"thread_start_us":24301,"threads_started":1} I0720 00:44:00.035236 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 2244 bytes on disk I0720 00:44:00.035749 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:00.036298 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000513 I0720 00:44:00.087378 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.051s user 0.015s sys 0.000s Metrics: {"bytes_written":10743,"cfile_init":1,"data dirs.queue_time_us":1045,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":525,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"rows_written":234} I0720 00:44:00.090853 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000527 I0720 00:44:00.169180 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.078s user 0.012s sys 0.003s Metrics: {"bytes_written":10539,"cfile_init":1,"data dirs.queue_time_us":228,"data dirs.run_cpu_time_us":172,"data dirs.run_wall_time_us":7613,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479,"lbm_writes_lt_1ms":23,"rows_written":225} I0720 00:44:00.173431 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000549 I0720 00:44:00.178862 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.164s user 0.010s sys 0.007s Metrics: {"bytes_written":11168,"cfile_init":1,"data dirs.queue_time_us":19425,"data dirs.run_cpu_time_us":152,"data dirs.run_wall_time_us":26941,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538,"lbm_writes_lt_1ms":23,"rows_written":251,"thread_start_us":86756,"threads_started":1} I0720 00:44:00.179945 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 2513 bytes on disk I0720 00:44:00.180493 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:00.184304 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000549 I0720 00:44:00.224526 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.040s user 0.013s sys 0.000s Metrics: {"bytes_written":10801,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":130,"data dirs.run_wall_time_us":638,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":517,"lbm_writes_lt_1ms":23,"rows_written":238} I0720 00:44:00.225584 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 2425 bytes on disk I0720 00:44:00.226130 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4} I0720 00:44:00.226756 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 1846 bytes on disk I0720 00:44:00.227165 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4} I0720 00:44:00.238710 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000561 I0720 00:44:00.338057 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.091s user 0.010s sys 0.007s Metrics: {"bytes_written":10962,"cfile_init":1,"data dirs.queue_time_us":79,"data dirs.run_cpu_time_us":190,"data dirs.run_wall_time_us":618,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1256,"lbm_writes_lt_1ms":23,"rows_written":244} I0720 00:44:00.339148 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 2452 bytes on disk I0720 00:44:00.339682 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4} I0720 00:44:00.342211 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 1700 bytes on disk I0720 00:44:00.342819 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:00.344578 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000297 I0720 00:44:00.382588 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.038s user 0.016s sys 0.000s Metrics: {"bytes_written":10218,"cfile_init":1,"data dirs.queue_time_us":84,"data dirs.run_cpu_time_us":176,"data dirs.run_wall_time_us":515,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"rows_written":215,"spinlock_wait_cycles":9472} I0720 00:44:00.387580 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41): perf score=0.990849 I0720 00:44:00.422956 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000315 I0720 00:44:00.429618 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.256s user 0.009s sys 0.006s Metrics: {"bytes_written":10741,"cfile_init":1,"data dirs.queue_time_us":40275,"data dirs.run_cpu_time_us":182,"data dirs.run_wall_time_us":6123,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":498,"lbm_writes_lt_1ms":23,"rows_written":234,"thread_start_us":102624,"threads_started":1} I0720 00:44:00.437155 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 2383 bytes on disk I0720 00:44:00.438031 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:00.443369 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 2347 bytes on disk I0720 00:44:00.448220 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:44:00.448930 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.026s user 0.011s sys 0.004s Metrics: {"bytes_written":10981,"cfile_init":1,"data dirs.queue_time_us":1962,"data dirs.run_cpu_time_us":158,"data dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":469,"lbm_writes_lt_1ms":23,"rows_written":244,"thread_start_us":1818,"threads_started":1} I0720 00:44:00.450119 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 2289 bytes on disk I0720 00:44:00.450542 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36): perf score=0.989564 I0720 00:44:00.450599 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:00.455127 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000292 I0720 00:44:00.476047 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.025s user 0.021s sys 0.004s Metrics: {"bytes_written":13294,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6308,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18524,"cfile_init":6,"data dirs.queue_time_us":2973,"data dirs.run_cpu_time_us":523,"data dirs.run_wall_time_us":3983,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":627,"lbm_reads_lt_1ms":50,"lbm_write_time_us":435,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":334,"thread_start_us":2527,"threads_started":2} I0720 00:44:00.477174 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 3289 bytes on disk I0720 00:44:00.477638 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} I0720 00:44:00.478276 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000326 I0720 00:44:00.500270 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.022s user 0.008s sys 0.009s Metrics: {"bytes_written":9566,"cfile_init":1,"data dirs.queue_time_us":72,"data dirs.run_cpu_time_us":122,"data dirs.run_wall_time_us":598,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":648,"lbm_writes_lt_1ms":23,"rows_written":190} I0720 00:44:00.501616 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 1964 bytes on disk I0720 00:44:00.502305 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4} I0720 00:44:00.503121 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.987903 I0720 00:44:00.520265 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.065s user 0.014s sys 0.004s Metrics: {"bytes_written":10021,"cfile_init":1,"data dirs.queue_time_us":4464,"data dirs.run_cpu_time_us":136,"data dirs.run_wall_time_us":543,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":661,"lbm_writes_lt_1ms":23,"rows_written":207} I0720 00:44:00.522053 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 2107 bytes on disk I0720 00:44:00.524814 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4} I0720 00:44:00.524945 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.022s user 0.013s sys 0.008s Metrics: {"bytes_written":11777,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5213,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15387,"cfile_init":5,"data dirs.queue_time_us":232,"data dirs.run_cpu_time_us":382,"data dirs.run_wall_time_us":810,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":847,"lbm_reads_lt_1ms":46,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":274} I0720 00:44:00.525856 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=1.010370 I0720 00:44:00.526243 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000335 I0720 00:44:00.546577 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.020s user 0.010s sys 0.010s Metrics: {"bytes_written":10220,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":457,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"rows_written":215} I0720 00:44:00.548018 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=1.002995 I0720 00:44:00.644148 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.256s user 0.017s sys 0.009s Metrics: {"bytes_written":12865,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6016,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17580,"cfile_init":7,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":94,"data dirs.run_wall_time_us":633,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":890,"lbm_reads_lt_1ms":54,"lbm_write_time_us":648,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":318} I0720 00:44:00.647639 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 3135 bytes on disk I0720 00:44:00.648506 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:00.649396 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000330 I0720 00:44:00.670382 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.144s user 0.057s sys 0.027s Metrics: {"bytes_written":12652,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5836,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17170,"cfile_init":5,"data dirs.queue_time_us":74885,"data dirs.run_cpu_time_us":733,"data dirs.run_wall_time_us":1177,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":5091,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":867,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":308,"thread_start_us":57921,"threads_started":1} I0720 00:44:00.679373 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 3077 bytes on disk I0720 00:44:00.680042 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:00.681731 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.133s user 0.021s sys 0.003s Metrics: {"bytes_written":12459,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5707,"cfile_cache_miss":26,"cfile_cache_miss_bytes":16805,"cfile_init":6,"data dirs.queue_time_us":1655,"data dirs.run_cpu_time_us":633,"data dirs.run_wall_time_us":938,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":983,"lbm_reads_lt_1ms":50,"lbm_write_time_us":732,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":301} I0720 00:44:00.684139 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000369 I0720 00:44:00.686329 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000350 I0720 00:44:00.710597 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.026s user 0.018s sys 0.005s Metrics: {"bytes_written":10829,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":453,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"rows_written":240} I0720 00:44:00.711943 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 3014 bytes on disk I0720 00:44:00.712512 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0720 00:44:00.713294 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41): perf score=0.991283 I0720 00:44:00.741655 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.028s user 0.017s sys 0.008s Metrics: {"bytes_written":14029,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6802,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19852,"cfile_init":6,"data dirs.queue_time_us":312,"data dirs.run_cpu_time_us":489,"data dirs.run_wall_time_us":873,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":996,"lbm_reads_lt_1ms":50,"lbm_write_time_us":773,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":361} I0720 00:44:00.742972 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 3522 bytes on disk I0720 00:44:00.743551 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:00.744341 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 2778 bytes on disk I0720 00:44:00.744911 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4} I0720 00:44:00.745777 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000382 I0720 00:44:00.768050 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.022s user 0.020s sys 0.000s Metrics: {"bytes_written":11139,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":96,"data dirs.run_wall_time_us":484,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"rows_written":249} I0720 00:44:00.769250 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 2528 bytes on disk I0720 00:44:00.769680 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0720 00:44:00.770268 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=1.014212 I0720 00:44:00.795442 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.025s user 0.021s sys 0.003s Metrics: {"bytes_written":14021,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6802,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19936,"cfile_init":5,"data dirs.queue_time_us":322,"data dirs.run_cpu_time_us":478,"data dirs.run_wall_time_us":865,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":834,"lbm_reads_lt_1ms":46,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":361} I0720 00:44:00.796754 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000391 I0720 00:44:00.827724 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.031s user 0.016s sys 0.001s Metrics: {"bytes_written":10594,"cfile_init":1,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":496,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"rows_written":230} I0720 00:44:00.828024 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.141s user 0.008s sys 0.014s Metrics: {"bytes_written":9711,"cfile_init":1,"data dirs.queue_time_us":8346,"data dirs.run_cpu_time_us":118,"data dirs.run_wall_time_us":484,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":738,"lbm_writes_lt_1ms":23,"rows_written":194} I0720 00:44:00.828917 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 3570 bytes on disk I0720 00:44:00.829490 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0720 00:44:00.830376 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 2344 bytes on disk I0720 00:44:00.830935 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:00.831678 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(c4890cfc794b469d8765701546219a19): perf score=1.013659 I0720 00:44:00.853688 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.022s user 0.021s sys 0.000s Metrics: {"bytes_written":13046,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6145,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17953,"cfile_init":5,"data dirs.queue_time_us":427,"data dirs.run_cpu_time_us":501,"data dirs.run_wall_time_us":920,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":823,"lbm_reads_lt_1ms":46,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":325,"spinlock_wait_cycles":10880} I0720 00:44:00.854614 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 3218 bytes on disk I0720 00:44:00.855084 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4} I0720 00:44:00.855929 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000399 I0720 00:44:00.883111 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.027s user 0.017s sys 0.004s Metrics: {"bytes_written":10736,"cfile_init":1,"data dirs.queue_time_us":60,"data dirs.run_cpu_time_us":120,"data dirs.run_wall_time_us":449,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":650,"lbm_writes_lt_1ms":23,"rows_written":234} I0720 00:44:00.884387 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946): perf score=1.010372 I0720 00:44:00.905006 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 1999 bytes on disk I0720 00:44:00.905649 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:00.906467 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(c4890cfc794b469d8765701546219a19): perf score=1.013412 I0720 00:44:00.932116 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.025s user 0.012s sys 0.011s Metrics: {"bytes_written":12406,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5690,"cfile_cache_miss":26,"cfile_cache_miss_bytes":16836,"cfile_init":5,"data dirs.queue_time_us":540,"data dirs.run_cpu_time_us":519,"data dirs.run_wall_time_us":870,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":990,"lbm_reads_lt_1ms":46,"lbm_write_time_us":669,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":300,"spinlock_wait_cycles":1280} I0720 00:44:00.933287 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000414 I0720 00:44:00.957021 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.072s user 0.021s sys 0.004s Metrics: {"bytes_written":13649,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6583,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19329,"cfile_init":6,"data dirs.queue_time_us":2454,"data dirs.run_cpu_time_us":552,"data dirs.run_wall_time_us":1081,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":938,"lbm_reads_lt_1ms":50,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":349} I0720 00:44:00.958446 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000377 I0720 00:44:00.963318 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.030s user 0.021s sys 0.004s Metrics: {"bytes_written":11366,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":126,"data dirs.run_wall_time_us":580,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1505,"lbm_writes_lt_1ms":23,"rows_written":259} I0720 00:44:00.964557 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 2996 bytes on disk I0720 00:44:00.965173 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:00.966027 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 2617 bytes on disk I0720 00:44:00.966650 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:00.967397 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946): perf score=1.010267 I0720 00:44:00.981166 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.331s user 0.016s sys 0.007s Metrics: {"bytes_written":11225,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":1017,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":8832,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":255} I0720 00:44:01.002511 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 3518 bytes on disk I0720 00:44:01.016500 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.049s user 0.021s sys 0.005s Metrics: {"bytes_written":14289,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7039,"cfile_cache_miss":26,"cfile_cache_miss_bytes":20615,"cfile_init":5,"data dirs.queue_time_us":6284,"data dirs.run_cpu_time_us":649,"data dirs.run_wall_time_us":1070,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1930,"lbm_reads_lt_1ms":46,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"mutex_wait_us":158,"num_input_rowsets":2,"rows_written":374} I0720 00:44:01.017899 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 3680 bytes on disk I0720 00:44:01.018749 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:01.019603 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000411 I0720 00:44:01.021308 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":175,"lbm_reads_lt_1ms":8} I0720 00:44:01.023052 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=1.009109 I0720 00:44:01.134821 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.115s user 0.017s sys 0.008s Metrics: {"bytes_written":10981,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":135,"data dirs.run_wall_time_us":886,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"rows_written":243} I0720 00:44:01.136204 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 2470 bytes on disk I0720 00:44:01.136878 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4} I0720 00:44:01.137697 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.997442 I0720 00:44:01.160763 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.137s user 0.020s sys 0.007s Metrics: {"bytes_written":13474,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6437,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18801,"cfile_init":5,"data dirs.queue_time_us":20925,"data dirs.run_cpu_time_us":608,"data dirs.run_wall_time_us":1108,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1057,"lbm_reads_lt_1ms":46,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":341,"thread_start_us":3086,"threads_started":1} I0720 00:44:01.162986 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 3391 bytes on disk I0720 00:44:01.163698 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:01.164520 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 1355 bytes on disk I0720 00:44:01.166695 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:01.167630 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000461 I0720 00:44:01.169705 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.032s user 0.024s sys 0.003s Metrics: {"bytes_written":13727,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6617,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19563,"cfile_init":5,"data dirs.queue_time_us":366,"data dirs.run_cpu_time_us":491,"data dirs.run_wall_time_us":867,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":934,"lbm_reads_lt_1ms":46,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":351} I0720 00:44:01.170986 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 3480 bytes on disk I0720 00:44:01.171551 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:01.172289 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000421 I0720 00:44:01.198750 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.240s user 0.020s sys 0.005s Metrics: {"bytes_written":11378,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":541,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"rows_written":257} I0720 00:44:01.199846 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 3450 bytes on disk I0720 00:44:01.200444 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4} I0720 00:44:01.201380 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=1.010777 I0720 00:44:01.235314 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.034s user 0.017s sys 0.005s Metrics: {"bytes_written":14273,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7005,"cfile_cache_miss":26,"cfile_cache_miss_bytes":20681,"cfile_init":6,"data dirs.queue_time_us":715,"data dirs.run_cpu_time_us":541,"data dirs.run_wall_time_us":939,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":4808,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":666,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":372} I0720 00:44:01.236171 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 3649 bytes on disk I0720 00:44:01.236598 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} I0720 00:44:01.237327 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000443 I0720 00:44:01.275732 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.038s user 0.022s sys 0.000s Metrics: {"bytes_written":11834,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":481,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"rows_written":277} I0720 00:44:01.277086 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 2763 bytes on disk I0720 00:44:01.277695 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:01.278503 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854): perf score=1.010474 I0720 00:44:01.308802 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.030s user 0.027s sys 0.000s Metrics: {"bytes_written":15521,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7898,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23010,"cfile_init":5,"data dirs.queue_time_us":319,"data dirs.run_cpu_time_us":508,"data dirs.run_wall_time_us":871,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":918,"lbm_reads_lt_1ms":46,"lbm_write_time_us":779,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":421} I0720 00:44:01.310111 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000457 I0720 00:44:01.338526 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.028s user 0.015s sys 0.008s Metrics: {"bytes_written":11580,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":487,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1083,"lbm_writes_lt_1ms":23,"rows_written":266} I0720 00:44:01.339884 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 4102 bytes on disk I0720 00:44:01.340446 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:01.341245 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d): perf score=1.014564 I0720 00:44:01.375011 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.033s user 0.023s sys 0.004s Metrics: {"bytes_written":15487,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7881,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23187,"cfile_init":6,"data dirs.queue_time_us":328,"data dirs.run_cpu_time_us":504,"data dirs.run_wall_time_us":888,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1336,"lbm_reads_lt_1ms":50,"lbm_write_time_us":752,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":420} I0720 00:44:01.376463 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 4082 bytes on disk I0720 00:44:01.377048 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:01.448401 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.276s user 0.016s sys 0.011s Metrics: {"bytes_written":11606,"cfile_init":1,"data dirs.queue_time_us":6101,"data dirs.run_cpu_time_us":120,"data dirs.run_wall_time_us":514,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"rows_written":268} I0720 00:44:01.459092 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 2682 bytes on disk I0720 00:44:01.459764 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4} I0720 00:44:01.460572 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d): perf score=1.007709 I0720 00:44:01.531110 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.363s user 0.014s sys 0.015s Metrics: {"bytes_written":11984,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":826,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":96333,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":21,"rows_written":282} I0720 00:44:01.532037 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 4027 bytes on disk I0720 00:44:01.532567 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":103,"lbm_reads_lt_1ms":8} I0720 00:44:01.533174 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d): perf score=1.007211 I0720 00:44:01.559546 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.099s user 0.016s sys 0.014s Metrics: {"bytes_written":15054,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7550,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22094,"cfile_init":5,"data dirs.queue_time_us":4559,"data dirs.run_cpu_time_us":673,"data dirs.run_wall_time_us":1099,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":699,"lbm_reads_lt_1ms":46,"lbm_write_time_us":750,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":402} I0720 00:44:01.561276 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 3934 bytes on disk I0720 00:44:01.562539 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:01.563455 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000465 I0720 00:44:01.628715 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000328 I0720 00:44:01.659045 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.030s user 0.014s sys 0.002s Metrics: {"bytes_written":8494,"cfile_init":1,"data dirs.queue_time_us":8261,"data dirs.run_cpu_time_us":145,"data dirs.run_wall_time_us":665,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"rows_written":147} I0720 00:44:01.660746 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36): perf score=1.004009 I0720 00:44:01.666524 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.103s user 0.018s sys 0.009s Metrics: {"bytes_written":11759,"cfile_init":1,"data dirs.queue_time_us":1681,"data dirs.run_cpu_time_us":129,"data dirs.run_wall_time_us":514,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":27653,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"rows_written":273} I0720 00:44:01.669832 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=1.012958 I0720 00:44:01.701472 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.036s user 0.026s sys 0.004s Metrics: {"bytes_written":17077,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8993,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26209,"cfile_init":6,"data dirs.queue_time_us":338,"data dirs.run_cpu_time_us":557,"data dirs.run_wall_time_us":1069,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1045,"lbm_reads_lt_1ms":50,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":481} I0720 00:44:01.702924 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 4649 bytes on disk I0720 00:44:01.704032 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:01.705024 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000335 I0720 00:44:01.741540 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.036s user 0.015s sys 0.000s Metrics: {"bytes_written":8485,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":490,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"rows_written":147} I0720 00:44:01.742838 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 1517 bytes on disk I0720 00:44:01.743531 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:01.744426 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4): perf score=1.003145 I0720 00:44:01.774868 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.242s user 0.022s sys 0.003s Metrics: {"bytes_written":14846,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7404,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21636,"cfile_init":5,"data dirs.queue_time_us":83734,"data dirs.run_cpu_time_us":694,"data dirs.run_wall_time_us":1177,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":741,"lbm_reads_lt_1ms":46,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"mutex_wait_us":133,"num_input_rowsets":2,"rows_written":394,"thread_start_us":54320,"threads_started":1} I0720 00:44:01.776268 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000622 I0720 00:44:01.778012 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.033s user 0.019s sys 0.008s Metrics: {"bytes_written":15508,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7898,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23174,"cfile_init":5,"data dirs.queue_time_us":328,"data dirs.run_cpu_time_us":532,"data dirs.run_wall_time_us":889,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":927,"lbm_reads_lt_1ms":46,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":421} I0720 00:44:01.779291 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 4133 bytes on disk I0720 00:44:01.780689 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4} I0720 00:44:01.781522 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000343 I0720 00:44:01.792889 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.123s user 0.028s sys 0.000s Metrics: {"bytes_written":15961,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8246,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24158,"cfile_init":6,"data dirs.queue_time_us":3313,"data dirs.run_cpu_time_us":563,"data dirs.run_wall_time_us":1001,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1010,"lbm_reads_lt_1ms":50,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":440} I0720 00:44:01.794255 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000489 I0720 00:44:01.821784 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.040s user 0.014s sys 0.000s Metrics: {"bytes_written":8492,"cfile_init":1,"data dirs.queue_time_us":62,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":446,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1824,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":146} I0720 00:44:01.825634 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 1544 bytes on disk I0720 00:44:01.826318 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:01.829221 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=1.006568 I0720 00:44:01.853108 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.077s user 0.022s sys 0.009s Metrics: {"bytes_written":13924,"cfile_init":1,"data dirs.queue_time_us":2291,"data dirs.run_cpu_time_us":115,"data dirs.run_wall_time_us":853,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":785,"lbm_writes_lt_1ms":23,"rows_written":359} I0720 00:44:01.859176 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854): perf score=1.010781 I0720 00:44:01.875697 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.046s user 0.028s sys 0.000s Metrics: {"bytes_written":16146,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8375,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24601,"cfile_init":5,"data dirs.queue_time_us":344,"data dirs.run_cpu_time_us":556,"data dirs.run_wall_time_us":971,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":976,"lbm_reads_lt_1ms":46,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":447} I0720 00:44:01.877022 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000324 I0720 00:44:01.895762 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":8654,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":1383,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":730,"lbm_writes_lt_1ms":23,"rows_written":153} I0720 00:44:01.897161 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41): perf score=1.011697 I0720 00:44:01.959713 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.062s user 0.024s sys 0.008s Metrics: {"bytes_written":17901,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9599,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27949,"cfile_init":6,"data dirs.queue_time_us":439,"data dirs.run_cpu_time_us":568,"data dirs.run_wall_time_us":1196,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":2796,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":780,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":514,"spinlock_wait_cycles":4864} I0720 00:44:01.961050 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000331 I0720 00:44:02.000957 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.039s user 0.012s sys 0.004s Metrics: {"bytes_written":8110,"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":142,"data dirs.run_wall_time_us":576,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"rows_written":133} I0720 00:44:02.003509 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=1.011913 I0720 00:44:02.083037 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.224s user 0.029s sys 0.002s Metrics: {"bytes_written":17293,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9139,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26585,"cfile_init":6,"data dirs.queue_time_us":3123,"data dirs.run_cpu_time_us":517,"data dirs.run_wall_time_us":3436,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":936,"lbm_reads_lt_1ms":50,"lbm_write_time_us":838,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":489} I0720 00:44:02.085705 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000680 I0720 00:44:02.093709 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.298s user 0.012s sys 0.016s Metrics: {"bytes_written":11802,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":523,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"rows_written":275} I0720 00:44:02.098359 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.132:0 --local_ip_for_outbound_sockets=127.0.58.132 --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:42541 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:02.100945 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.095s user 0.031s sys 0.000s Metrics: {"bytes_written":17343,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9234,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26932,"cfile_init":6,"data dirs.queue_time_us":2327,"data dirs.run_cpu_time_us":601,"data dirs.run_wall_time_us":1114,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1211,"lbm_reads_lt_1ms":50,"lbm_write_time_us":790,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":494} I0720 00:44:02.101857 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 4983 bytes on disk I0720 00:44:02.104064 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 4294 bytes on disk I0720 00:44:02.104642 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2269,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0720 00:44:02.105150 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:02.105258 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 4800 bytes on disk I0720 00:44:02.105870 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 2779 bytes on disk I0720 00:44:02.106338 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:02.106459 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:02.107089 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000355 I0720 00:44:02.107245 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36): perf score=1.011583 I0720 00:44:02.126196 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.040s user 0.026s sys 0.004s Metrics: {"bytes_written":14254,"cfile_init":1,"compiler_manager_pool.queue_time_us":18167,"compiler_manager_pool.run_cpu_time_us":28,"compiler_manager_pool.run_wall_time_us":28,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":2166,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":667,"lbm_writes_lt_1ms":23,"rows_written":371} I0720 00:44:02.127555 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.996000 I0720 00:44:02.132745 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.025s user 0.009s sys 0.002s Metrics: {"bytes_written":9096,"cfile_init":1,"data dirs.queue_time_us":71,"data dirs.run_cpu_time_us":123,"data dirs.run_wall_time_us":473,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"rows_written":171} I0720 00:44:02.134001 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(c4890cfc794b469d8765701546219a19): perf score=1.008503 I0720 00:44:02.139611 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.032s user 0.025s sys 0.006s Metrics: {"bytes_written":17476,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9285,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27105,"cfile_init":5,"data dirs.queue_time_us":517,"data dirs.run_cpu_time_us":558,"data dirs.run_wall_time_us":928,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1439,"lbm_reads_lt_1ms":46,"lbm_write_time_us":657,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":497} I0720 00:44:02.140530 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 4802 bytes on disk I0720 00:44:02.141098 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:02.141646 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000571 I0720 00:44:02.156253 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.028s user 0.024s sys 0.004s Metrics: {"bytes_written":17479,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9302,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27048,"cfile_init":6,"data dirs.queue_time_us":1012,"data dirs.run_cpu_time_us":494,"data dirs.run_wall_time_us":894,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":912,"lbm_reads_lt_1ms":50,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"mutex_wait_us":23,"num_input_rowsets":2,"rows_written":498,"spinlock_wait_cycles":3456} I0720 00:44:02.157475 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 4837 bytes on disk I0720 00:44:02.158215 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:02.159013 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000633 I0720 00:44:02.167598 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.026s user 0.024s sys 0.000s Metrics: {"bytes_written":12596,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":90,"data dirs.run_wall_time_us":600,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701,"lbm_writes_lt_1ms":23,"rows_written":307} I0720 00:44:02.168596 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41): perf score=1.011915 I0720 00:44:02.182529 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.044s user 0.021s sys 0.001s Metrics: {"bytes_written":17355,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9268,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26936,"cfile_init":6,"data dirs.queue_time_us":6259,"data dirs.run_cpu_time_us":483,"data dirs.run_wall_time_us":826,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":637,"lbm_reads_lt_1ms":50,"lbm_write_time_us":589,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":496,"spinlock_wait_cycles":1792} I0720 00:44:02.183686 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000361 I0720 00:44:02.200943 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.042s user 0.021s sys 0.007s Metrics: {"bytes_written":13900,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":106,"data dirs.run_wall_time_us":441,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":658,"lbm_writes_lt_1ms":23,"rows_written":359} I0720 00:44:02.202145 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.033s user 0.026s sys 0.007s Metrics: {"bytes_written":18579,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10088,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29394,"cfile_init":6,"data dirs.queue_time_us":467,"data dirs.run_cpu_time_us":487,"data dirs.run_wall_time_us":822,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":959,"lbm_reads_lt_1ms":50,"lbm_write_time_us":670,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":541,"spinlock_wait_cycles":7168} I0720 00:44:02.202323 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946): perf score=1.006443 I0720 00:44:02.203132 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000565 I0720 00:44:02.201931 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.018s user 0.005s sys 0.004s Metrics: {"bytes_written":9173,"cfile_init":1,"data dirs.queue_time_us":64,"data dirs.run_cpu_time_us":124,"data dirs.run_wall_time_us":546,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":454,"lbm_writes_lt_1ms":23,"rows_written":175,"spinlock_wait_cycles":14720} I0720 00:44:02.207170 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 4802 bytes on disk I0720 00:44:02.210536 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:02.211285 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 4372 bytes on disk I0720 00:44:02.211731 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:02.213497 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 1790 bytes on disk I0720 00:44:02.213945 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:02.214635 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946): perf score=1.010605 I0720 00:44:02.230579 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.027s user 0.022s sys 0.004s Metrics: {"bytes_written":12447,"cfile_init":1,"data dirs.queue_time_us":49,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":452,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":658,"lbm_writes_lt_1ms":23,"rows_written":301} I0720 00:44:02.231830 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854): perf score=1.010105 I0720 00:44:02.232532 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.030s user 0.028s sys 0.001s Metrics: {"bytes_written":18098,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9779,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28313,"cfile_init":6,"data dirs.queue_time_us":445,"data dirs.run_cpu_time_us":508,"data dirs.run_wall_time_us":939,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":917,"lbm_reads_lt_1ms":50,"lbm_write_time_us":772,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":524} I0720 00:44:02.233750 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000624 I0720 00:44:02.262306 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.028s user 0.028s sys 0.000s Metrics: {"bytes_written":13298,"cfile_init":1,"data dirs.queue_time_us":58,"data dirs.run_cpu_time_us":120,"data dirs.run_wall_time_us":566,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"rows_written":336} I0720 00:44:02.264478 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=1.011171 I0720 00:44:02.265506 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.033s user 0.024s sys 0.008s Metrics: {"bytes_written":18158,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9818,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28676,"cfile_init":6,"data dirs.queue_time_us":349,"data dirs.run_cpu_time_us":496,"data dirs.run_wall_time_us":864,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":968,"lbm_reads_lt_1ms":50,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":526} I0720 00:44:02.266762 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 5257 bytes on disk I0720 00:44:02.265836 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.051s user 0.026s sys 0.001s Metrics: {"bytes_written":18098,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9779,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28303,"cfile_init":5,"data dirs.queue_time_us":3067,"data dirs.run_cpu_time_us":534,"data dirs.run_wall_time_us":903,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":773,"lbm_reads_lt_1ms":46,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":524,"spinlock_wait_cycles":2944} I0720 00:44:02.267279 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:02.267938 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 5128 bytes on disk I0720 00:44:02.268010 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000297 I0720 00:44:02.268437 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:02.269124 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000511 I0720 00:44:02.288115 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.020s user 0.015s sys 0.002s Metrics: {"bytes_written":8362,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":482,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"rows_written":143} I0720 00:44:02.289356 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 5109 bytes on disk I0720 00:44:02.289928 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4} I0720 00:44:02.291823 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=1.010599 I0720 00:44:02.293009 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.024s user 0.016s sys 0.004s Metrics: {"bytes_written":11427,"cfile_init":1,"data dirs.queue_time_us":54,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":559,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":684,"lbm_writes_lt_1ms":23,"rows_written":261} I0720 00:44:02.294209 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4): perf score=1.013214 I0720 00:44:02.298786 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.034s user 0.022s sys 0.008s Metrics: {"bytes_written":17934,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9610,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27942,"cfile_init":6,"data dirs.queue_time_us":314,"data dirs.run_cpu_time_us":477,"data dirs.run_wall_time_us":1297,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":902,"lbm_reads_lt_1ms":50,"lbm_write_time_us":795,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":515} I0720 00:44:02.299947 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 5109 bytes on disk I0720 00:44:02.300496 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:02.301177 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 4978 bytes on disk I0720 00:44:02.301620 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:02.302268 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 4736 bytes on disk I0720 00:44:02.302731 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4} I0720 00:44:02.303472 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000593 I0720 00:44:02.326243 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.032s user 0.017s sys 0.013s Metrics: {"bytes_written":17361,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9251,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27065,"cfile_init":6,"data dirs.queue_time_us":357,"data dirs.run_cpu_time_us":549,"data dirs.run_wall_time_us":992,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":989,"lbm_reads_lt_1ms":50,"lbm_write_time_us":662,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":495} I0720 00:44:02.327050 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.035s user 0.022s sys 0.012s Metrics: {"bytes_written":17934,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9611,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27909,"cfile_init":6,"data dirs.queue_time_us":536,"data dirs.run_cpu_time_us":351,"data dirs.run_wall_time_us":725,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":921,"lbm_reads_lt_1ms":50,"lbm_write_time_us":734,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":515,"spinlock_wait_cycles":1280} I0720 00:44:02.327546 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 4820 bytes on disk I0720 00:44:02.328016 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4} I0720 00:44:02.328212 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 4978 bytes on disk I0720 00:44:02.328662 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=0.000502 I0720 00:44:02.328675 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0720 00:44:02.329336 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000293 I0720 00:44:02.331359 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.028s user 0.024s sys 0.000s Metrics: {"bytes_written":12278,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":477,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"rows_written":296} I0720 00:44:02.332486 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 3849 bytes on disk I0720 00:44:02.333045 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:02.333781 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36): perf score=1.011671 I0720 00:44:02.347337 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":7428,"cfile_init":1,"data dirs.queue_time_us":55,"data dirs.run_cpu_time_us":104,"data dirs.run_wall_time_us":445,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"rows_written":105} I0720 00:44:02.348548 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.994318 I0720 00:44:02.351521 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.023s user 0.009s sys 0.011s Metrics: {"bytes_written":10015,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":88,"data dirs.run_wall_time_us":474,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"rows_written":207} I0720 00:44:02.352727 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac): perf score=1.011313 I0720 00:44:02.371839 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.038s user 0.032s sys 0.003s Metrics: {"bytes_written":18723,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10194,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29632,"cfile_init":6,"data dirs.queue_time_us":332,"data dirs.run_cpu_time_us":435,"data dirs.run_wall_time_us":822,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1395,"lbm_reads_lt_1ms":50,"lbm_write_time_us":629,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":547} I0720 00:44:02.373091 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000597 I0720 00:44:02.380095 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.031s user 0.022s sys 0.008s Metrics: {"bytes_written":18158,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9818,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28502,"cfile_init":6,"data dirs.queue_time_us":350,"data dirs.run_cpu_time_us":528,"data dirs.run_wall_time_us":1481,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":866,"lbm_reads_lt_1ms":50,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":526,"spinlock_wait_cycles":1920} I0720 00:44:02.381347 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000290 I0720 00:44:02.387473 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.034s user 0.025s sys 0.008s Metrics: {"bytes_written":17934,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9611,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28021,"cfile_init":6,"data dirs.queue_time_us":349,"data dirs.run_cpu_time_us":536,"data dirs.run_wall_time_us":924,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":946,"lbm_reads_lt_1ms":50,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":515} I0720 00:44:02.388916 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac): 4978 bytes on disk I0720 00:44:02.389595 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(0a7ad36f80f44067ba0087a2768e26ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:02.390398 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000434 I0720 00:44:02.398524 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":7665,"cfile_init":1,"data dirs.queue_time_us":38,"data dirs.run_cpu_time_us":85,"data dirs.run_wall_time_us":632,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":23,"rows_written":115} I0720 00:44:02.399782 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d): perf score=1.000827 I0720 00:44:02.400985 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.028s user 0.013s sys 0.011s Metrics: {"bytes_written":11361,"cfile_init":1,"data dirs.queue_time_us":49,"data dirs.run_cpu_time_us":87,"data dirs.run_wall_time_us":424,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":684,"lbm_writes_lt_1ms":23,"rows_written":257} I0720 00:44:02.402212 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 5307 bytes on disk I0720 00:44:02.402777 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:02.403515 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4): perf score=1.013215 I0720 00:44:02.414535 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.024s user 0.015s sys 0.008s Metrics: {"bytes_written":9736,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":99,"data dirs.run_wall_time_us":485,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2261,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":196} I0720 00:44:02.415729 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(c4890cfc794b469d8765701546219a19): perf score=1.008971 I0720 00:44:02.433210 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.033s user 0.024s sys 0.008s Metrics: {"bytes_written":18390,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9976,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28920,"cfile_init":6,"data dirs.queue_time_us":529,"data dirs.run_cpu_time_us":453,"data dirs.run_wall_time_us":892,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":917,"lbm_reads_lt_1ms":50,"lbm_write_time_us":872,"lbm_writes_lt_1ms":23,"mutex_wait_us":29,"num_input_rowsets":2,"rows_written":535,"spinlock_wait_cycles":5632} I0720 00:44:02.434471 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 5171 bytes on disk I0720 00:44:02.435056 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:02.435823 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 5128 bytes on disk I0720 00:44:02.436305 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:02.436177 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.032s user 0.020s sys 0.009s Metrics: {"bytes_written":17361,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9251,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27053,"cfile_init":6,"data dirs.queue_time_us":1278,"data dirs.run_cpu_time_us":476,"data dirs.run_wall_time_us":865,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":883,"lbm_reads_lt_1ms":50,"lbm_write_time_us":703,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":495} I0720 00:44:02.437249 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(c4890cfc794b469d8765701546219a19): perf score=0.000583 I0720 00:44:02.443866 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.028s user 0.019s sys 0.008s Metrics: {"bytes_written":17355,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9263,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26967,"cfile_init":6,"data dirs.queue_time_us":456,"data dirs.run_cpu_time_us":501,"data dirs.run_wall_time_us":889,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":865,"lbm_reads_lt_1ms":50,"lbm_write_time_us":651,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":496,"spinlock_wait_cycles":9600} I0720 00:44:02.444813 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(73ea05088cb24563bcf684d6199dd946): perf score=0.000411 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:02.446542 4980 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:02.446843 4980 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:02.449982 4980 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:02.450214 4980 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:02.450268 4980 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:02.450362 4980 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.132 I0720 00:44:02.457285 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.012s user 0.006s sys 0.004s Metrics: {"bytes_written":8533,"cfile_init":1,"data dirs.queue_time_us":71,"data dirs.run_cpu_time_us":123,"data dirs.run_wall_time_us":495,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":434,"lbm_writes_lt_1ms":23,"rows_written":150} I0720 00:44:02.458319 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946): perf score=1.010916 I0720 00:44:02.460973 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.022s user 0.018s sys 0.001s Metrics: {"bytes_written":11167,"cfile_init":1,"data dirs.queue_time_us":77,"data dirs.run_cpu_time_us":122,"data dirs.run_wall_time_us":483,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635,"lbm_writes_lt_1ms":23,"rows_written":252} I0720 00:44:02.461496 4980 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --webserver_interface=127.0.58.132 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:42541 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.132 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:02.462371 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(c4890cfc794b469d8765701546219a19): perf score=1.012427 I0720 00:44:02.471699 4980 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 82516us I0720 00:44:02.472579 4980 fs_manager.cc:263] Metadata directory not provided I0720 00:44:02.472671 4980 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:02.472997 4980 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:02.473186 4980 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:02.477329 4980 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/instance: uuid: "e23cbbcbc8444eec834170f2d501c129" format_stamp: "Formatted at 2019-07-20 00:44:02 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:02.477998 4980 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-1/instance: uuid: "e23cbbcbc8444eec834170f2d501c129" format_stamp: "Formatted at 2019-07-20 00:44:02 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:02.478463 4980 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-2/instance: uuid: "e23cbbcbc8444eec834170f2d501c129" format_stamp: "Formatted at 2019-07-20 00:44:02 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:02.478912 4980 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/wal/instance: uuid: "e23cbbcbc8444eec834170f2d501c129" format_stamp: "Formatted at 2019-07-20 00:44:02 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:02.481847 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.023s user 0.022s sys 0.001s Metrics: {"bytes_written":18098,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9779,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28285,"cfile_init":6,"data dirs.queue_time_us":515,"data dirs.run_cpu_time_us":454,"data dirs.run_wall_time_us":808,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":702,"lbm_reads_lt_1ms":50,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":524,"spinlock_wait_cycles":10368} I0720 00:44:02.482669 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946): 5109 bytes on disk I0720 00:44:02.483088 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ea05088cb24563bcf684d6199dd946) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4} I0720 00:44:02.483570 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000375 I0720 00:44:02.491489 4980 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.010s sys 0.002s I0720 00:44:02.493232 4980 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:02.493906 4980 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:02.494549 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.032s user 0.025s sys 0.004s Metrics: {"bytes_written":17355,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9262,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26952,"cfile_init":6,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":472,"data dirs.run_wall_time_us":823,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":960,"lbm_reads_lt_1ms":50,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":496} I0720 00:44:02.495873 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 4820 bytes on disk I0720 00:44:02.496462 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:02.497150 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000587 I0720 00:44:02.498603 4980 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0720 00:44:02.498721 4980 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/wal uuid: "e23cbbcbc8444eec834170f2d501c129" format_stamp: "Formatted at 2019-07-20 00:44:02 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:02.498891 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.015s user 0.007s sys 0.007s Metrics: {"bytes_written":8490,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":525,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":516,"lbm_writes_lt_1ms":23,"rows_written":147,"spinlock_wait_cycles":4480} I0720 00:44:02.498950 4980 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:02.499953 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.987295 I0720 00:44:02.511745 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.014s user 0.012s sys 0.001s Metrics: {"bytes_written":10400,"cfile_init":1,"data dirs.queue_time_us":66,"data dirs.run_cpu_time_us":126,"data dirs.run_wall_time_us":479,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"rows_written":223} I0720 00:44:02.512787 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41): perf score=1.011788 I0720 00:44:02.529215 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.029s user 0.020s sys 0.007s Metrics: {"bytes_written":17479,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9302,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27076,"cfile_init":6,"data dirs.queue_time_us":494,"data dirs.run_cpu_time_us":505,"data dirs.run_wall_time_us":1053,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":910,"lbm_reads_lt_1ms":50,"lbm_write_time_us":644,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":498,"spinlock_wait_cycles":25344} I0720 00:44:02.530254 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 4837 bytes on disk I0720 00:44:02.531322 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:02.531919 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 4802 bytes on disk I0720 00:44:02.532313 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:02.532800 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000301 I0720 00:44:02.535208 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.022s user 0.017s sys 0.005s Metrics: {"bytes_written":18579,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10088,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29192,"cfile_init":6,"data dirs.queue_time_us":533,"data dirs.run_cpu_time_us":493,"data dirs.run_wall_time_us":855,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":791,"lbm_reads_lt_1ms":50,"lbm_write_time_us":494,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":541,"spinlock_wait_cycles":1408} I0720 00:44:02.536219 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000432 I0720 00:44:02.546165 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.013s user 0.013s sys 0.000s Metrics: {"bytes_written":8112,"cfile_init":1,"data dirs.queue_time_us":69,"data dirs.run_cpu_time_us":129,"data dirs.run_wall_time_us":486,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":534,"lbm_writes_lt_1ms":23,"rows_written":133} I0720 00:44:02.547194 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 1410 bytes on disk I0720 00:44:02.547703 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:02.548498 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d): perf score=1.001382 I0720 00:44:02.551877 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":9369,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":494,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":23,"rows_written":182} I0720 00:44:02.552887 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 5257 bytes on disk I0720 00:44:02.553381 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:02.554052 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19): 4802 bytes on disk I0720 00:44:02.554481 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(c4890cfc794b469d8765701546219a19) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4} I0720 00:44:02.555105 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=1.011921 I0720 00:44:02.555235 4980 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:02.559257 4980 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:02.559321 4980 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:02.578743 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.023s user 0.017s sys 0.004s Metrics: {"bytes_written":18105,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9762,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28396,"cfile_init":6,"data dirs.queue_time_us":701,"data dirs.run_cpu_time_us":484,"data dirs.run_wall_time_us":885,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":662,"lbm_reads_lt_1ms":50,"lbm_write_time_us":677,"lbm_writes_lt_1ms":23,"mutex_wait_us":128,"num_input_rowsets":2,"rows_written":523} I0720 00:44:02.579810 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d): perf score=0.000291 I0720 00:44:02.582533 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.033s user 0.030s sys 0.001s Metrics: {"bytes_written":18390,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9981,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28997,"cfile_init":5,"data dirs.queue_time_us":947,"data dirs.run_cpu_time_us":520,"data dirs.run_wall_time_us":896,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":927,"lbm_reads_lt_1ms":46,"lbm_write_time_us":719,"lbm_writes_lt_1ms":23,"mutex_wait_us":231,"num_input_rowsets":2,"rows_written":535} I0720 00:44:02.583703 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 5171 bytes on disk I0720 00:44:02.584218 4425 maintenance_manager.cc:532] P 506b8430b3284890b468e7ef55b6ca79: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:02.594836 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":8318,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":508,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":568,"lbm_writes_lt_1ms":23,"rows_written":141} I0720 00:44:02.595880 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 5079 bytes on disk I0720 00:44:02.596400 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:02.596926 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d): perf score=1.001641 I0720 00:44:02.620002 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.023s user 0.019s sys 0.004s Metrics: {"bytes_written":18390,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9981,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28973,"cfile_init":6,"data dirs.queue_time_us":837,"data dirs.run_cpu_time_us":503,"data dirs.run_wall_time_us":1404,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":874,"lbm_reads_lt_1ms":50,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"mutex_wait_us":19,"num_input_rowsets":2,"rows_written":535,"spinlock_wait_cycles":42496} I0720 00:44:02.621067 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d): 5171 bytes on disk I0720 00:44:02.621939 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(73ac7512c8584fe3ad887ffef570df8d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:02.673369 4980 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.132:46503 I0720 00:44:02.674461 4980 webserver.cc:374] Webserver started at http://127.0.58.132:38831/ using document root and password file I0720 00:44:02.675094 4980 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb I0720 00:44:02.679514 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 4980 W0720 00:44:02.688678 4527 generic_service.cc:128] {username='slave'} at 127.0.0.1:51180 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:02.688784 4527 generic_service.cc:148] {username='slave'} at 127.0.0.1:51180 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:02.689241 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(9ef65edd881743f691ebb903dad80c36): perf score=0.000286 I0720 00:44:02.693368 5104 heartbeater.cc:324] Connected to a master server at 127.0.58.190:42541 I0720 00:44:02.693562 5104 heartbeater.cc:405] Registering TS with master... I0720 00:44:02.694021 5104 heartbeater.cc:454] Master 127.0.58.190:42541 requested a full tablet report, sending... I0720 00:44:02.695067 4362 ts_manager.cc:161] Registered new tserver with Master: e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503) I0720 00:44:02.695916 4362 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.132:58167 I0720 00:44:02.702806 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.013s user 0.011s sys 0.002s Metrics: {"bytes_written":6398,"cfile_init":1,"data dirs.queue_time_us":81,"data dirs.run_cpu_time_us":116,"data dirs.run_wall_time_us":497,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":643,"lbm_writes_lt_1ms":23,"rows_written":66} I0720 00:44:02.703698 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 749 bytes on disk I0720 00:44:02.704332 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:02.704936 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36): perf score=0.988868 W0720 00:44:02.706274 4652 generic_service.cc:128] {username='slave'} at 127.0.0.1:52192 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:02.707463 4652 generic_service.cc:148] {username='slave'} at 127.0.0.1:52192 changed flags via RPC: enable_tablet_copy from 'true' to 'false' W0720 00:44:02.714180 4776 generic_service.cc:128] {username='slave'} at 127.0.0.1:35922 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:02.714290 4776 generic_service.cc:148] {username='slave'} at 127.0.0.1:35922 changed flags via RPC: enable_tablet_copy from 'true' to 'false' W0720 00:44:02.722357 5101 generic_service.cc:128] {username='slave'} at 127.0.0.1:51614 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:02.722461 5101 generic_service.cc:148] {username='slave'} at 127.0.0.1:51614 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:02.724074 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 4398 I0720 00:44:02.725216 4527 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:02.728335 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.023s user 0.023s sys 0.000s Metrics: {"bytes_written":18723,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10200,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29514,"cfile_init":5,"data dirs.queue_time_us":686,"data dirs.run_cpu_time_us":580,"data dirs.run_wall_time_us":962,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":803,"lbm_reads_lt_1ms":46,"lbm_write_time_us":1565,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":15,"num_input_rowsets":2,"rows_written":547,"spinlock_wait_cycles":17152} I0720 00:44:02.729457 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36): 5307 bytes on disk I0720 00:44:02.729921 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(9ef65edd881743f691ebb903dad80c36) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} I0720 00:44:02.864184 4531 maintenance_manager.cc:327] P 506b8430b3284890b468e7ef55b6ca79: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000333 I0720 00:44:02.873018 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.000283 I0720 00:44:02.877544 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 4398 I0720 00:44:02.883239 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: FlushMRSOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.010s user 0.005s sys 0.004s Metrics: {"bytes_written":5617,"cfile_init":1,"data dirs.queue_time_us":70,"data dirs.run_cpu_time_us":106,"data dirs.run_wall_time_us":499,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":529,"lbm_writes_lt_1ms":23,"rows_written":37} I0720 00:44:02.884169 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854): perf score=0.943987 I0720 00:44:02.908795 234 disk_failure-itest.cc:273] Restarting server with injected errors... I0720 00:44:02.909335 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.129:36375 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_port=33417 --webserver_interface=127.0.58.129 --tserver_master_addrs=127.0.58.190:42541 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 --cfile_inject_corruption=1.0 with env {} I0720 00:44:02.910287 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: CompactRowSetsOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.026s user 0.024s sys 0.002s Metrics: {"bytes_written":18158,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9818,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28392,"cfile_init":6,"data dirs.queue_time_us":702,"data dirs.run_cpu_time_us":327,"data dirs.run_wall_time_us":789,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":767,"lbm_reads_lt_1ms":50,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":526,"spinlock_wait_cycles":18688} I0720 00:44:02.911190 4656 maintenance_manager.cc:327] P a29a463e4c2e4f61a400dde426c625e8: Scheduling UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854): 5128 bytes on disk I0720 00:44:02.911723 4550 maintenance_manager.cc:532] P a29a463e4c2e4f61a400dde426c625e8: UndoDeltaBlockGCOp(19920b309dd84a01b2cc74ddbbd10854) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} W0720 00:44:02.941457 4668 consensus_peers.cc:458] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:02.981243 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(27285c1e3c36466f9bce8450be7500d4): perf score=0.000344 W0720 00:44:02.984849 4544 consensus_peers.cc:458] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:02.991354 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.010s user 0.008s sys 0.001s Metrics: {"bytes_written":6615,"cfile_init":1,"data dirs.queue_time_us":77,"data dirs.run_cpu_time_us":129,"data dirs.run_wall_time_us":560,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":505,"lbm_writes_lt_1ms":23,"rows_written":74} I0720 00:44:02.992198 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4): perf score=1.011663 W0720 00:44:03.007922 4544 consensus_peers.cc:458] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.011415 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.019s user 0.016s sys 0.003s Metrics: {"bytes_written":17361,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9251,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26899,"cfile_init":6,"data dirs.queue_time_us":693,"data dirs.run_cpu_time_us":517,"data dirs.run_wall_time_us":881,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":638,"lbm_reads_lt_1ms":50,"lbm_write_time_us":471,"lbm_writes_lt_1ms":23,"mutex_wait_us":20,"num_input_rowsets":2,"rows_written":495,"spinlock_wait_cycles":16256} I0720 00:44:03.012295 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.000331 W0720 00:44:03.013098 4668 consensus_peers.cc:458] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.020330 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.008s user 0.005s sys 0.000s Metrics: {"bytes_written":6006,"cfile_init":1,"data dirs.queue_time_us":68,"data dirs.run_cpu_time_us":117,"data dirs.run_wall_time_us":516,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":396,"lbm_writes_lt_1ms":23,"rows_written":51,"spinlock_wait_cycles":13440} I0720 00:44:03.021214 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4): 4820 bytes on disk I0720 00:44:03.021669 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(27285c1e3c36466f9bce8450be7500d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:03.022243 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b): perf score=0.971195 I0720 00:44:03.040370 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.018s user 0.017s sys 0.001s Metrics: {"bytes_written":17479,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9302,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26900,"cfile_init":6,"data dirs.queue_time_us":499,"data dirs.run_cpu_time_us":360,"data dirs.run_wall_time_us":753,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":632,"lbm_reads_lt_1ms":50,"lbm_write_time_us":441,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":498,"spinlock_wait_cycles":44800} I0720 00:44:03.041368 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b): 4837 bytes on disk I0720 00:44:03.041885 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(ca6a2a19cbbe4648a8a76de1e43c405b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:03.042436 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(5758df4920ce4bbab0424048f13fea41): perf score=0.000318 I0720 00:44:03.049927 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.007s user 0.006s sys 0.000s Metrics: {"bytes_written":5360,"cfile_init":1,"data dirs.queue_time_us":58,"data dirs.run_cpu_time_us":123,"data dirs.run_wall_time_us":495,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"rows_written":27,"spinlock_wait_cycles":1280} I0720 00:44:03.050837 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 376 bytes on disk I0720 00:44:03.051262 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0720 00:44:03.051719 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41): perf score=0.861418 W0720 00:44:03.068850 4544 consensus_peers.cc:458] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:03.069401 5114 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:03.069633 5114 flags.cc:404] Enabled unsafe flag: --never_fsync=true I0720 00:44:03.071163 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.019s user 0.017s sys 0.001s Metrics: {"bytes_written":18579,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10088,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29122,"cfile_init":5,"data dirs.queue_time_us":535,"data dirs.run_cpu_time_us":290,"data dirs.run_wall_time_us":663,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":661,"lbm_reads_lt_1ms":46,"lbm_write_time_us":531,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":541} W0720 00:44:03.072046 5114 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 I0720 00:44:03.072131 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.000297 W0720 00:44:03.072168 5114 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:03.072213 5114 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:03.072291 5114 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:44:03.076684 5114 tablet_server_main.cc:78] Tablet server non-default flags: --cfile_inject_corruption=1 --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:36375 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=33417 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:42541 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:03.079540 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: FlushMRSOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.007s user 0.006s sys 0.000s Metrics: {"bytes_written":5399,"cfile_init":1,"data dirs.queue_time_us":60,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":531,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":407,"lbm_writes_lt_1ms":23,"rows_written":29} I0720 00:44:03.080473 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41): 5257 bytes on disk I0720 00:44:03.080921 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(5758df4920ce4bbab0424048f13fea41) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} I0720 00:44:03.081506 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787): perf score=0.997629 W0720 00:44:03.081753 4668 consensus_peers.cc:458] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.084777 5114 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 83016us I0720 00:44:03.085430 5114 fs_manager.cc:263] Metadata directory not provided I0720 00:44:03.085517 5114 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:03.096747 5114 fs_manager.cc:399] Time spent opening directory manager: real 0.010s user 0.009s sys 0.000s I0720 00:44:03.098304 5114 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:03.098925 5114 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:03.105087 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: CompactRowSetsOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.023s user 0.017s sys 0.003s Metrics: {"bytes_written":18105,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9762,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28100,"cfile_init":6,"data dirs.queue_time_us":675,"data dirs.run_cpu_time_us":492,"data dirs.run_wall_time_us":938,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":719,"lbm_reads_lt_1ms":50,"lbm_write_time_us":414,"lbm_writes_lt_1ms":23,"mutex_wait_us":10,"num_input_rowsets":2,"rows_written":523,"spinlock_wait_cycles":11264} I0720 00:44:03.105924 4780 maintenance_manager.cc:327] P fc86e9a330584f00b998288875c0586f: Scheduling UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787): 5079 bytes on disk I0720 00:44:03.106384 4674 maintenance_manager.cc:532] P fc86e9a330584f00b998288875c0586f: UndoDeltaBlockGCOp(7869d31b2ece4c38b8cf73ba177d2787) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4} W0720 00:44:03.107513 4544 consensus_peers.cc:458] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.114508 5114 fs_manager.cc:419] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s I0720 00:44:03.114591 5114 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal uuid: "506b8430b3284890b468e7ef55b6ca79" format_stamp: "Formatted at 2019-07-20 00:43:57 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:03.115572 5114 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-2/data Total live blocks: 55 Total live bytes: 233728 Total live bytes (after alignment): 393216 Total number of LBM containers: 16 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) W0720 00:44:03.142896 4544 consensus_peers.cc:458] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.146466 5114 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go W0720 00:44:03.147004 4668 consensus_peers.cc:458] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:36375: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.148301 5114 ts_tablet_manager.cc:345] Loading tablet metadata (0/10 complete) I0720 00:44:03.165967 5114 ts_tablet_manager.cc:357] Loaded tablet metadata (10 total tablets, 10 live tablets) I0720 00:44:03.166047 5114 ts_tablet_manager.cc:363] Registering tablets (0/10 complete) I0720 00:44:03.169096 5133 tablet_bootstrap.cc:439] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:44:03.170624 5134 tablet_bootstrap.cc:439] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:44:03.172588 5135 tablet_bootstrap.cc:439] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. W0720 00:44:03.173957 5134 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:44:03.174008 5134 fault_injection.cc:45] THIS SERVER MAY CRASH! E0720 00:44:03.174157 5134 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000128 E0720 00:44:03.174224 5135 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000236 E0720 00:44:03.174662 5135 tablet.cc:285] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 0000000000000236: Checksum does not match: 672874039 vs expected 672874039 E0720 00:44:03.174643 5134 tablet.cc:285] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 0000000000000128: Checksum does not match: 396618722 vs expected 396618722 E0720 00:44:03.174878 5133 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000188 E0720 00:44:03.175082 5133 tablet.cc:285] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 0000000000000188: Checksum does not match: 62407076 vs expected 62407076 E0720 00:44:03.175341 5134 ts_tablet_manager.cc:1080] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000128: Checksum does not match: 396618722 vs expected 396618722 E0720 00:44:03.175339 5135 ts_tablet_manager.cc:1080] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000236: Checksum does not match: 672874039 vs expected 672874039 E0720 00:44:03.175345 5133 ts_tablet_manager.cc:1080] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000188: Checksum does not match: 62407076 vs expected 62407076 I0720 00:44:03.175495 5135 ts_tablet_manager.cc:1059] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0720 00:44:03.175523 5133 ts_tablet_manager.cc:1059] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s I0720 00:44:03.175520 5134 ts_tablet_manager.cc:1059] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s I0720 00:44:03.175637 5133 tablet_replica.cc:273] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.175637 5135 tablet_replica.cc:273] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.175729 5134 tablet_replica.cc:273] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.175782 5133 raft_consensus.cc:2147] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.175803 5135 raft_consensus.cc:2147] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.177094 5133 raft_consensus.cc:2174] T 0a7ad36f80f44067ba0087a2768e26ac P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.175842 5134 raft_consensus.cc:2147] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.177198 5134 raft_consensus.cc:2174] T 7869d31b2ece4c38b8cf73ba177d2787 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.177445 5133 tablet_bootstrap.cc:439] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:44:03.177486 5134 tablet_bootstrap.cc:439] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. I0720 00:44:03.177098 5135 raft_consensus.cc:2174] T 73ac7512c8584fe3ad887ffef570df8d P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.178879 5135 tablet_bootstrap.cc:439] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. E0720 00:44:03.178879 5134 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000140 E0720 00:44:03.178879 5133 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000176 E0720 00:44:03.179150 5133 tablet.cc:285] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 0000000000000176: Checksum does not match: 2919875488 vs expected 2919875488 E0720 00:44:03.179145 5134 tablet.cc:285] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 0000000000000140: Checksum does not match: 3072083029 vs expected 3072083029 E0720 00:44:03.179370 5133 ts_tablet_manager.cc:1080] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000176: Checksum does not match: 2919875488 vs expected 2919875488 E0720 00:44:03.179368 5134 ts_tablet_manager.cc:1080] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000140: Checksum does not match: 3072083029 vs expected 3072083029 I0720 00:44:03.179530 5133 ts_tablet_manager.cc:1059] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:03.179533 5134 ts_tablet_manager.cc:1059] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s I0720 00:44:03.179616 5133 tablet_replica.cc:273] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.179636 5134 tablet_replica.cc:273] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.179749 5133 raft_consensus.cc:2147] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.179792 5134 raft_consensus.cc:2147] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.179829 5133 raft_consensus.cc:2174] T 27285c1e3c36466f9bce8450be7500d4 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.179853 5134 raft_consensus.cc:2174] T 9ef65edd881743f691ebb903dad80c36 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! E0720 00:44:03.180336 5135 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000164 E0720 00:44:03.180493 5135 tablet.cc:285] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 0000000000000164: Checksum does not match: 3177807480 vs expected 3177807480 E0720 00:44:03.180687 5135 ts_tablet_manager.cc:1080] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000164: Checksum does not match: 3177807480 vs expected 3177807480 I0720 00:44:03.180749 5135 ts_tablet_manager.cc:1059] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:03.180804 5135 tablet_replica.cc:273] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.180858 5135 raft_consensus.cc:2147] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.180914 5135 raft_consensus.cc:2174] T 19920b309dd84a01b2cc74ddbbd10854 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.181295 5135 tablet_bootstrap.cc:439] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. E0720 00:44:03.182458 5135 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000152 E0720 00:44:03.182603 5135 tablet.cc:285] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(2): Corruption: failed to init CFileReader for block 0000000000000152: Checksum does not match: 1478762943 vs expected 1478762943 E0720 00:44:03.182773 5135 ts_tablet_manager.cc:1080] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000152: Checksum does not match: 1478762943 vs expected 1478762943 I0720 00:44:03.182835 5135 ts_tablet_manager.cc:1059] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I0720 00:44:03.182884 5135 tablet_replica.cc:273] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.182941 5135 raft_consensus.cc:2147] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.182996 5135 raft_consensus.cc:2174] T 5758df4920ce4bbab0424048f13fea41 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.183199 5135 tablet_bootstrap.cc:439] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. E0720 00:44:03.184298 5135 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000212 E0720 00:44:03.184437 5135 tablet.cc:285] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 0000000000000212: Checksum does not match: 3440600655 vs expected 3440600655 E0720 00:44:03.184625 5135 ts_tablet_manager.cc:1080] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000212: Checksum does not match: 3440600655 vs expected 3440600655 I0720 00:44:03.184689 5135 ts_tablet_manager.cc:1059] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0720 00:44:03.184737 5135 tablet_replica.cc:273] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.184790 5135 raft_consensus.cc:2147] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.184846 5135 raft_consensus.cc:2174] T 73ea05088cb24563bcf684d6199dd946 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.185163 5135 tablet_bootstrap.cc:439] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. E0720 00:44:03.186300 5135 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000200 E0720 00:44:03.186436 5135 tablet.cc:285] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 0000000000000200: Checksum does not match: 3565727781 vs expected 3565727781 E0720 00:44:03.186609 5135 ts_tablet_manager.cc:1080] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000200: Checksum does not match: 3565727781 vs expected 3565727781 I0720 00:44:03.186668 5135 ts_tablet_manager.cc:1059] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:03.186725 5135 tablet_replica.cc:273] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.186776 5135 raft_consensus.cc:2147] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.186830 5135 raft_consensus.cc:2174] T c4890cfc794b469d8765701546219a19 P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.186971 5114 ts_tablet_manager.cc:377] Registered 10 tablets I0720 00:44:03.187060 5135 tablet_bootstrap.cc:439] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Bootstrap starting. E0720 00:44:03.188251 5135 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000224 I0720 00:44:03.188338 5135 ts_tablet_manager.cc:1475] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: failing tablet E0720 00:44:03.188606 5135 tablet.cc:285] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Failed to open rowset RowSet(4): Corruption: failed to init CFileReader for block 0000000000000224: Checksum does not match: 1683009991 vs expected 1683009991 E0720 00:44:03.188792 5135 ts_tablet_manager.cc:1080] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Tablet failed to bootstrap: Corruption: failed to init CFileReader for block 0000000000000224: Checksum does not match: 1683009991 vs expected 1683009991 I0720 00:44:03.188874 5135 ts_tablet_manager.cc:1059] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0720 00:44:03.188958 5135 tablet_replica.cc:273] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79: stopping tablet replica I0720 00:44:03.189014 5135 raft_consensus.cc:2147] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:03.189090 5135 raft_consensus.cc:2174] T ca6a2a19cbbe4648a8a76de1e43c405b P 506b8430b3284890b468e7ef55b6ca79 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:03.213259 5114 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:36375 I0720 00:44:03.214325 5114 webserver.cc:374] Webserver started at http://127.0.58.129:33417/ using document root and password file I0720 00:44:03.215056 5114 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:44:03.218024 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 5114 I0720 00:44:03.224982 234 disk_failure-itest.cc:256] Currently has 10 failed tablets I0720 00:44:03.225131 234 disk_failure-itest.cc:234] Resetting error injection flags I0720 00:44:03.234668 5240 heartbeater.cc:324] Connected to a master server at 127.0.58.190:42541 I0720 00:44:03.234859 5240 heartbeater.cc:405] Registering TS with master... W0720 00:44:03.235139 5237 generic_service.cc:128] {username='slave'} at 127.0.0.1:51190 forcing change of non-runtime-safe flag env_inject_eio I0720 00:44:03.235260 5237 generic_service.cc:148] {username='slave'} at 127.0.0.1:51190 changed flags via RPC: env_inject_eio from '0' to '0.0' I0720 00:44:03.235363 5240 heartbeater.cc:454] Master 127.0.58.190:42541 requested a full tablet report, sending... W0720 00:44:03.236593 5237 generic_service.cc:128] {username='slave'} at 127.0.0.1:51190 forcing change of non-runtime-safe flag cfile_inject_corruption I0720 00:44:03.236667 5237 generic_service.cc:148] {username='slave'} at 127.0.0.1:51190 changed flags via RPC: cfile_inject_corruption from '1' to '0.0' I0720 00:44:03.237144 4362 ts_manager.cc:161] Re-registered known tserver with Master: 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375) W0720 00:44:03.237493 5237 generic_service.cc:128] {username='slave'} at 127.0.0.1:51190 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:03.237570 5237 generic_service.cc:148] {username='slave'} at 127.0.0.1:51190 changed flags via RPC: enable_tablet_copy from 'true' to 'true' W0720 00:44:03.237816 4362 catalog_manager.cc:3988] Tablet 0a7ad36f80f44067ba0087a2768e26ac (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000188: Checksum does not match: 62407076 vs expected 62407076 W0720 00:44:03.237910 4362 catalog_manager.cc:3988] Tablet 7869d31b2ece4c38b8cf73ba177d2787 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000128: Checksum does not match: 396618722 vs expected 396618722 W0720 00:44:03.237987 4362 catalog_manager.cc:3988] Tablet 19920b309dd84a01b2cc74ddbbd10854 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000164: Checksum does not match: 3177807480 vs expected 3177807480 W0720 00:44:03.238041 4362 catalog_manager.cc:3988] Tablet 73ac7512c8584fe3ad887ffef570df8d (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000236: Checksum does not match: 672874039 vs expected 672874039 W0720 00:44:03.238128 4362 catalog_manager.cc:3988] Tablet 9ef65edd881743f691ebb903dad80c36 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000140: Checksum does not match: 3072083029 vs expected 3072083029 W0720 00:44:03.238189 4362 catalog_manager.cc:3988] Tablet 27285c1e3c36466f9bce8450be7500d4 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000176: Checksum does not match: 2919875488 vs expected 2919875488 W0720 00:44:03.238256 4362 catalog_manager.cc:3988] Tablet 5758df4920ce4bbab0424048f13fea41 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000152: Checksum does not match: 1478762943 vs expected 1478762943 W0720 00:44:03.238307 4362 catalog_manager.cc:3988] Tablet 73ea05088cb24563bcf684d6199dd946 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000212: Checksum does not match: 3440600655 vs expected 3440600655 W0720 00:44:03.238369 4362 catalog_manager.cc:3988] Tablet ca6a2a19cbbe4648a8a76de1e43c405b (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000224: Checksum does not match: 1683009991 vs expected 1683009991 W0720 00:44:03.238417 4362 catalog_manager.cc:3988] Tablet c4890cfc794b469d8765701546219a19 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000200: Checksum does not match: 3565727781 vs expected 3565727781 I0720 00:44:03.239634 4362 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:60217 I0720 00:44:03.268227 4668 consensus_queue.cc:544] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed W0720 00:44:03.268368 4668 consensus_peers.cc:458] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f -> Peer 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Couldn't send request to peer 506b8430b3284890b468e7ef55b6ca79. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: failed to init CFileReader for block 0000000000000164: Checksum does not match: 3177807480 vs expected 3177807480. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.276213 4358 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 19920b309dd84a01b2cc74ddbbd10854 (attempt 1) I0720 00:44:03.277881 4734 consensus_queue.cc:206] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.279582 4611 raft_consensus.cc:1184] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:03.280076 5062 consensus_queue.cc:984] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.282033 5263 raft_consensus.cc:2792] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.282464 4611 raft_consensus.cc:2792] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.285763 4358 catalog_manager.cc:4067] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.313452 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 19920b309dd84a01b2cc74ddbbd10854 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.316666 4358 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 19920b309dd84a01b2cc74ddbbd10854 (attempt 1) I0720 00:44:03.317883 4734 consensus_queue.cc:206] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.319815 4611 raft_consensus.cc:1184] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:03.320283 5062 consensus_queue.cc:984] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:03.322330 5062 raft_consensus.cc:2792] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.324185 4611 raft_consensus.cc:2792] T 19920b309dd84a01b2cc74ddbbd10854 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.325598 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 19920b309dd84a01b2cc74ddbbd10854 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.328044 4358 catalog_manager.cc:4067] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0720 00:44:03.330243 4670 consensus_peers.cc:458] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19920b309dd84a01b2cc74ddbbd10854. This is attempt 1: this message will repeat every 5th retry. Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 537ffdd6efb847e991c0d499bb244e71 | 127.0.58.190:42541 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 506b8430b3284890b468e7ef55b6ca79 | 127.0.58.129:36375 | HEALTHY | a29a463e4c2e4f61a400dde426c625e8 | 127.0.58.130:43527 | HEALTHY | e23cbbcbc8444eec834170f2d501c129 | 127.0.58.132:46503 | HEALTHY | fc86e9a330584f00b998288875c0586f | 127.0.58.131:42343 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:36375 enable_tablet_copy | false | unsafe | 127.0.58.130:43527, 127.0.58.132:46503, 127.0.58.131:42343 enable_tablet_copy | true | unsafe | 127.0.58.129:36375 env_inject_eio | 0 | hidden | 127.0.58.129:36375 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:36375 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:43527 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:42343 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:46503 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:36375 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:43527 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:42343 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:46503 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 73ac7512c8584fe3ad887ffef570df8d of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000236: Checksum does not match: 672874039 vs expected 672874039 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 5758df4920ce4bbab0424048f13fea41 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000152: Checksum does not match: 1478762943 vs expected 1478762943 fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = a29a463e4c2e4f61a400dde426c625e8 B = 506b8430b3284890b468e7ef55b6ca79 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Tablet 9ef65edd881743f691ebb903dad80c36 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000140: Checksum does not match: 3072083029 vs expected 3072083029 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 27285c1e3c36466f9bce8450be7500d4 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000176: Checksum does not match: 2919875488 vs expected 2919875488 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 73ea05088cb24563bcf684d6199dd946 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000212: Checksum does not match: 3440600655 vs expected 3440600655 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 19920b309dd84a01b2cc74ddbbd10854 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000164: Checksum does not match: 3177807480 vs expected 3177807480 fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = a29a463e4c2e4f61a400dde426c625e8 B = 506b8430b3284890b468e7ef55b6ca79 C = fc86e9a330584f00b998288875c0586f D = e23cbbcbc8444eec834170f2d501c129 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A C* | 1 | 106 | Yes B | [config not available] | | | C | A C* D~ | 1 | 107 | Yes Tablet ca6a2a19cbbe4648a8a76de1e43c405b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000224: Checksum does not match: 1683009991 vs expected 1683009991 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 0a7ad36f80f44067ba0087a2768e26ac of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000188: Checksum does not match: 62407076 vs expected 62407076 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 7869d31b2ece4c38b8cf73ba177d2787 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000128: Checksum does not match: 396618722 vs expected 396618722 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet c4890cfc794b469d8765701546219a19 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000200: Checksum does not match: 3565727781 vs expected 3565727781 fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = a29a463e4c2e4f61a400dde426c625e8 B = 506b8430b3284890b468e7ef55b6ca79 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 0 | 0 | 10 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | e23cbbcbc8444eec834170f2d501c129 | 127.0.58.132:46503 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:03.345481 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:44:03.376880 4544 consensus_queue.cc:544] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.378397 4362 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 27285c1e3c36466f9bce8450be7500d4 (attempt 1) I0720 00:44:03.379752 4611 consensus_queue.cc:206] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.381140 4734 raft_consensus.cc:1184] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch) I0720 00:44:03.381582 5278 consensus_queue.cc:984] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s I0720 00:44:03.383316 5279 raft_consensus.cc:2792] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.383756 4734 raft_consensus.cc:2792] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.385772 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 27285c1e3c36466f9bce8450be7500d4 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.386595 4362 catalog_manager.cc:4067] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index -1 to 104, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.388939 4362 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 27285c1e3c36466f9bce8450be7500d4 (attempt 1) I0720 00:44:03.389931 4611 consensus_queue.cc:206] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.391458 4734 raft_consensus.cc:1184] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:03.391937 5279 consensus_queue.cc:984] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:03.393702 5278 raft_consensus.cc:2792] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 104 to 105, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.394207 4734 raft_consensus.cc:2792] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 104 to 105, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.396250 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27285c1e3c36466f9bce8450be7500d4 with cas_config_opid_index 104: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.397214 4362 catalog_manager.cc:4067] T 27285c1e3c36466f9bce8450be7500d4 P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index 104 to 105, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } W0720 00:44:03.398672 4546 consensus_peers.cc:458] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27285c1e3c36466f9bce8450be7500d4. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.422211 4668 consensus_queue.cc:544] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.424182 4358 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 5758df4920ce4bbab0424048f13fea41 (attempt 1) I0720 00:44:03.425117 4734 consensus_queue.cc:206] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.426437 4611 raft_consensus.cc:1184] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:03.426877 5277 consensus_queue.cc:984] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.428573 5263 raft_consensus.cc:2792] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.429028 4611 raft_consensus.cc:2792] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.430740 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 5758df4920ce4bbab0424048f13fea41 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.431937 4358 catalog_manager.cc:4067] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.434669 4358 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5758df4920ce4bbab0424048f13fea41 (attempt 1) I0720 00:44:03.435612 4734 consensus_queue.cc:206] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } W0720 00:44:03.437234 4670 consensus_peers.cc:458] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5758df4920ce4bbab0424048f13fea41. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.437317 4611 raft_consensus.cc:1184] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:03.437736 5263 consensus_queue.cc:984] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:03.439332 5277 raft_consensus.cc:2792] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.439834 4611 raft_consensus.cc:2792] T 5758df4920ce4bbab0424048f13fea41 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.442057 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5758df4920ce4bbab0424048f13fea41 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.443024 4358 catalog_manager.cc:4067] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:03.477458 4668 consensus_queue.cc:544] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.479482 4362 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 73ac7512c8584fe3ad887ffef570df8d (attempt 1) I0720 00:44:03.480435 4734 consensus_queue.cc:206] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.482102 4611 raft_consensus.cc:1184] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:03.482545 5263 consensus_queue.cc:984] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.484181 5277 raft_consensus.cc:2792] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.484647 4611 raft_consensus.cc:2792] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.486291 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 73ac7512c8584fe3ad887ffef570df8d with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.487151 4362 catalog_manager.cc:4067] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.489456 4362 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 73ac7512c8584fe3ad887ffef570df8d (attempt 1) I0720 00:44:03.490744 4734 consensus_queue.cc:206] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.491255 4362 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 73ac7512c8584fe3ad887ffef570df8d (attempt 1) I0720 00:44:03.492012 4611 raft_consensus.cc:1184] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:03.492430 5277 consensus_queue.cc:984] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s W0720 00:44:03.492683 4670 consensus_peers.cc:458] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73ac7512c8584fe3ad887ffef570df8d. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.492683 4345 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 73ac7512c8584fe3ad887ffef570df8d with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER failed with leader fc86e9a330584f00b998288875c0586f (127.0.58.131:42343) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } }. Pending config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.492791 4345 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 73ac7512c8584fe3ad887ffef570df8d with cas_config_opid_index 106 with a delay of 32 ms (attempt = 1) I0720 00:44:03.494170 5263 raft_consensus.cc:2792] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.494643 4611 raft_consensus.cc:2792] T 73ac7512c8584fe3ad887ffef570df8d P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.496767 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 73ac7512c8584fe3ad887ffef570df8d with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.498441 4362 catalog_manager.cc:4067] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:03.525961 4345 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 73ac7512c8584fe3ad887ffef570df8d with cas_config_opid_index 106: aborting the task: latest config opid_index 107; task opid_index 106 I0720 00:44:03.559332 4544 consensus_queue.cc:544] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.561759 4362 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 73ea05088cb24563bcf684d6199dd946 (attempt 1) Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 537ffdd6efb847e991c0d499bb244e71 | 127.0.58.190:42541 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 506b8430b3284890b468e7ef55b6ca79 | 127.0.58.129:36375 | HEALTHY | a29a463e4c2e4f61a400dde426c625e8 | 127.0.58.130:43527 | HEALTHY | e23cbbcbc8444eec834170f2d501c129 | 127.0.58.132:46503 | HEALTHY | fc86e9a330584f00b998288875c0586f | 127.0.58.131:42343 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:36375 enable_tablet_copy | false | unsafe | 127.0.58.130:43527, 127.0.58.132:46503, 127.0.58.131:42343 enable_tablet_copy | true | unsafe | 127.0.58.129:36375 env_inject_eio | 0 | hidden | 127.0.58.129:36375 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:36375 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:43527 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:42343 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:46503 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:36375 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:43527 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:42343 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:46503 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 9ef65edd881743f691ebb903dad80c36 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000140: Checksum does not match: 3072083029 vs expected 3072083029 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f I0720 00:44:03.562841 4611 consensus_queue.cc:206] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 73ea05088cb24563bcf684d6199dd946 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000212: Checksum does not match: 3440600655 vs expected 3440600655 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet ca6a2a19cbbe4648a8a76de1e43c405b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000224: Checksum does not match: 1683009991 vs expected 1683009991 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet 0a7ad36f80f44067ba0087a2768e26ac of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000188: Checksum does not match: 62407076 vs expected 62407076 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet 7869d31b2ece4c38b8cf73ba177d2787 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000128: Checksum does not match: 396618722 vs expected 396618722 a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING [LEADER] All reported replicas are: A = 506b8430b3284890b468e7ef55b6ca79 B = a29a463e4c2e4f61a400dde426c625e8 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet c4890cfc794b469d8765701546219a19 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING a29a463e4c2e4f61a400dde426c625e8 (127.0.58.130:43527): RUNNING [LEADER] 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): not running State: FAILED Data state: TABLET_DATA_READY Last status: Corruption: failed to init CFileReader for block 0000000000000200: Checksum does not match: 3565727781 vs expected 3565727781 fc86e9a330584f00b998288875c0586f (127.0.58.131:42343): RUNNING All reported replicas are: A = a29a463e4c2e4f61a400dde426c625e8 B = 506b8430b3284890b468e7ef55b6ca79 C = fc86e9a330584f00b998288875c0586f The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 4 | 0 | 6 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 4 First Quartile | 6 Median | 10 Third Quartile | 10 Maximum | 10 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:03.566505 4734 raft_consensus.cc:1184] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:03.566921 5279 consensus_queue.cc:984] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.569085 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:44:03.569265 4952 raft_consensus.cc:2792] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.569741 4734 raft_consensus.cc:2792] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.571656 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 73ea05088cb24563bcf684d6199dd946 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.572511 4362 catalog_manager.cc:4067] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.574932 4362 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 73ea05088cb24563bcf684d6199dd946 (attempt 1) I0720 00:44:03.575901 4611 consensus_queue.cc:206] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.577352 4734 raft_consensus.cc:1184] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) W0720 00:44:03.577517 4546 consensus_peers.cc:458] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73ea05088cb24563bcf684d6199dd946. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.577822 4952 consensus_queue.cc:984] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:03.579447 5279 raft_consensus.cc:2792] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.579984 4735 raft_consensus.cc:2792] T 73ea05088cb24563bcf684d6199dd946 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.581766 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 73ea05088cb24563bcf684d6199dd946 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.582458 4362 catalog_manager.cc:4067] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:03.602735 4668 consensus_queue.cc:544] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.605067 4358 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 0a7ad36f80f44067ba0087a2768e26ac (attempt 1) I0720 00:44:03.606549 4735 consensus_queue.cc:206] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.610683 4611 raft_consensus.cc:1184] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:03.611215 5277 consensus_queue.cc:984] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:03.613206 5263 raft_consensus.cc:2792] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.613657 4611 raft_consensus.cc:2792] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.615510 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 0a7ad36f80f44067ba0087a2768e26ac with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.616233 4358 catalog_manager.cc:4067] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index -1 to 105, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.618643 4358 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 0a7ad36f80f44067ba0087a2768e26ac (attempt 1) I0720 00:44:03.619853 4735 consensus_queue.cc:206] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.621431 4611 raft_consensus.cc:1184] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) W0720 00:44:03.621718 4670 consensus_peers.cc:458] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a7ad36f80f44067ba0087a2768e26ac. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.621830 5277 consensus_queue.cc:984] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.623468 5263 raft_consensus.cc:2792] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.623889 4611 raft_consensus.cc:2792] T 0a7ad36f80f44067ba0087a2768e26ac P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.625653 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0a7ad36f80f44067ba0087a2768e26ac with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.626730 4358 catalog_manager.cc:4067] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index 105 to 106, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:03.628823 4544 consensus_queue.cc:544] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.629029 4544 consensus_queue.cc:544] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.631096 4358 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet ca6a2a19cbbe4648a8a76de1e43c405b (attempt 1) I0720 00:44:03.631259 4358 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 9ef65edd881743f691ebb903dad80c36 (attempt 1) I0720 00:44:03.632052 4611 consensus_queue.cc:206] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.632052 4612 consensus_queue.cc:206] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.633610 4735 raft_consensus.cc:1184] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:03.634161 4735 raft_consensus.cc:1184] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:03.634219 5278 consensus_queue.cc:984] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.634567 4952 consensus_queue.cc:984] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.635838 5278 raft_consensus.cc:2792] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.636312 5279 raft_consensus.cc:2792] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.636356 4734 raft_consensus.cc:2792] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.636857 4735 raft_consensus.cc:2792] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.638671 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 9ef65edd881743f691ebb903dad80c36 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.639389 4358 catalog_manager.cc:4067] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.641125 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet ca6a2a19cbbe4648a8a76de1e43c405b with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.641814 4358 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9ef65edd881743f691ebb903dad80c36 (attempt 1) I0720 00:44:03.641860 4362 catalog_manager.cc:4067] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index -1 to 106, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.642839 4611 consensus_queue.cc:206] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.644242 4358 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet ca6a2a19cbbe4648a8a76de1e43c405b (attempt 1) I0720 00:44:03.644459 4735 raft_consensus.cc:1184] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:03.644938 4952 consensus_queue.cc:984] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s W0720 00:44:03.645351 4546 consensus_peers.cc:458] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ef65edd881743f691ebb903dad80c36. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.645531 4611 consensus_queue.cc:206] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.646631 5278 raft_consensus.cc:2792] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.646988 4734 raft_consensus.cc:1184] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) W0720 00:44:03.647050 4546 consensus_peers.cc:458] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca6a2a19cbbe4648a8a76de1e43c405b. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.647471 5279 consensus_queue.cc:984] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:03.648145 4668 consensus_queue.cc:544] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.649228 5279 raft_consensus.cc:2792] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.649696 4358 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 7869d31b2ece4c38b8cf73ba177d2787 (attempt 1) I0720 00:44:03.649706 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9ef65edd881743f691ebb903dad80c36 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.650041 4735 raft_consensus.cc:2792] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.650903 4734 consensus_queue.cc:206] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.651048 4736 raft_consensus.cc:2792] T 9ef65edd881743f691ebb903dad80c36 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.652259 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca6a2a19cbbe4648a8a76de1e43c405b with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.653918 4358 catalog_manager.cc:4067] T ca6a2a19cbbe4648a8a76de1e43c405b P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.655226 4362 catalog_manager.cc:4067] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index 106 to 107, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:03.655797 4611 raft_consensus.cc:1184] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch) I0720 00:44:03.656940 5263 consensus_queue.cc:984] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s I0720 00:44:03.660504 5062 raft_consensus.cc:2792] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.661058 4611 raft_consensus.cc:2792] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index -1 to 104, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.662554 4544 consensus_queue.cc:544] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: The tablet replica hosted on peer 506b8430b3284890b468e7ef55b6ca79 has failed I0720 00:44:03.663271 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 7869d31b2ece4c38b8cf73ba177d2787 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.664171 4358 catalog_manager.cc:4067] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index -1 to 104, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } } I0720 00:44:03.664481 4362 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet c4890cfc794b469d8765701546219a19 (attempt 1) I0720 00:44:03.665477 4611 consensus_queue.cc:206] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } I0720 00:44:03.667131 4358 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7869d31b2ece4c38b8cf73ba177d2787 (attempt 1) I0720 00:44:03.667376 4736 raft_consensus.cc:1184] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:03.667783 5313 consensus_queue.cc:984] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:03.668087 4736 consensus_queue.cc:206] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.669386 4611 raft_consensus.cc:1184] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Refusing update from remote peer fc86e9a330584f00b998288875c0586f: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) W0720 00:44:03.669715 4670 consensus_peers.cc:458] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7869d31b2ece4c38b8cf73ba177d2787. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.669772 5062 consensus_queue.cc:984] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:03.670208 5312 raft_consensus.cc:2792] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.670787 4735 raft_consensus.cc:2792] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.671460 5263 raft_consensus.cc:2792] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index 104 to 105, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.671998 4611 raft_consensus.cc:2792] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index 104 to 105, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.673511 4345 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet c4890cfc794b469d8765701546219a19 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:03.674460 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7869d31b2ece4c38b8cf73ba177d2787 with cas_config_opid_index 104: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.674919 4358 catalog_manager.cc:4067] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f reported cstate change: config changed from index -1 to 105, VOTER 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } } I0720 00:44:03.677110 4362 catalog_manager.cc:4067] T 7869d31b2ece4c38b8cf73ba177d2787 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index 104 to 105, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "fc86e9a330584f00b998288875c0586f" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.679114 4362 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c4890cfc794b469d8765701546219a19 (attempt 1) I0720 00:44:03.680058 4611 consensus_queue.cc:206] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } I0720 00:44:03.681586 4735 raft_consensus.cc:1184] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Refusing update from remote peer a29a463e4c2e4f61a400dde426c625e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) W0720 00:44:03.681710 4546 consensus_peers.cc:458] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Couldn't send request to peer e23cbbcbc8444eec834170f2d501c129. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c4890cfc794b469d8765701546219a19. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:03.681973 5313 consensus_queue.cc:984] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:03.683502 5312 raft_consensus.cc:2792] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.683986 4735 raft_consensus.cc:2792] T c4890cfc794b469d8765701546219a19 P fc86e9a330584f00b998288875c0586f [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } } } I0720 00:44:03.685719 4345 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c4890cfc794b469d8765701546219a19 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:03.686687 4362 catalog_manager.cc:4067] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 reported cstate change: config changed from index 105 to 106, NON_VOTER e23cbbcbc8444eec834170f2d501c129 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "a29a463e4c2e4f61a400dde426c625e8" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "a29a463e4c2e4f61a400dde426c625e8" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc86e9a330584f00b998288875c0586f" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 42343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e23cbbcbc8444eec834170f2d501c129" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 46503 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:03.697728 5104 heartbeater.cc:446] Master 127.0.58.190:42541 was elected leader, sending a full tablet report... W0720 00:44:03.801618 5312 consensus_peers.cc:251] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 537ffdd6efb847e991c0d499bb244e71 | 127.0.58.190:42541 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 506b8430b3284890b468e7ef55b6ca79 | 127.0.58.129:36375 | HEALTHY | a29a463e4c2e4f61a400dde426c625e8 | 127.0.58.130:43527 | HEALTHY | e23cbbcbc8444eec834170f2d501c129 | 127.0.58.132:46503 | HEALTHY | fc86e9a330584f00b998288875c0586f | 127.0.58.131:42343 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:36375 enable_tablet_copy | false | unsafe | 127.0.58.130:43527, 127.0.58.132:46503, 127.0.58.131:42343 enable_tablet_copy | true | unsafe | 127.0.58.129:36375 env_inject_eio | 0 | hidden | 127.0.58.129:36375 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:36375 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:43527 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:42343 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:46503 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:36375 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:43527 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:42343 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:46503 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 10 | 10 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 506b8430b3284890b468e7ef55b6ca79 | 127.0.58.129:36375 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:44:03.864776 234 log_verifier.cc:121] Checking tablet 0a7ad36f80f44067ba0087a2768e26ac W0720 00:44:03.894836 5277 consensus_peers.cc:251] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:03.921432 5277 consensus_peers.cc:251] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:03.936940 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet 0a7ad36f80f44067ba0087a2768e26ac I0720 00:44:03.937124 234 log_verifier.cc:121] Checking tablet 19920b309dd84a01b2cc74ddbbd10854 W0720 00:44:03.967905 5312 consensus_peers.cc:251] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:03.973947 5277 consensus_peers.cc:251] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.004166 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 19920b309dd84a01b2cc74ddbbd10854 I0720 00:44:04.004361 234 log_verifier.cc:121] Checking tablet 27285c1e3c36466f9bce8450be7500d4 I0720 00:44:04.069384 234 log_verifier.cc:172] Verified matching terms for 105 ops in tablet 27285c1e3c36466f9bce8450be7500d4 I0720 00:44:04.069552 234 log_verifier.cc:121] Checking tablet 5758df4920ce4bbab0424048f13fea41 W0720 00:44:04.082644 5312 consensus_peers.cc:251] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.094427 5277 consensus_peers.cc:251] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.097651 5277 consensus_peers.cc:251] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.135129 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 5758df4920ce4bbab0424048f13fea41 I0720 00:44:04.135293 234 log_verifier.cc:121] Checking tablet 73ac7512c8584fe3ad887ffef570df8d W0720 00:44:04.152009 5312 consensus_peers.cc:251] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.202128 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 73ac7512c8584fe3ad887ffef570df8d I0720 00:44:04.202294 234 log_verifier.cc:121] Checking tablet 73ea05088cb24563bcf684d6199dd946 I0720 00:44:04.242061 5240 heartbeater.cc:446] Master 127.0.58.190:42541 was elected leader, sending a full tablet report... W0720 00:44:04.244076 4358 catalog_manager.cc:3988] Tablet 0a7ad36f80f44067ba0087a2768e26ac (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000188: Checksum does not match: 62407076 vs expected 62407076 W0720 00:44:04.244199 4358 catalog_manager.cc:3988] Tablet 7869d31b2ece4c38b8cf73ba177d2787 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000128: Checksum does not match: 396618722 vs expected 396618722 W0720 00:44:04.244285 4358 catalog_manager.cc:3988] Tablet 19920b309dd84a01b2cc74ddbbd10854 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000164: Checksum does not match: 3177807480 vs expected 3177807480 W0720 00:44:04.244364 4358 catalog_manager.cc:3988] Tablet 73ac7512c8584fe3ad887ffef570df8d (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000236: Checksum does not match: 672874039 vs expected 672874039 W0720 00:44:04.244463 4358 catalog_manager.cc:3988] Tablet 9ef65edd881743f691ebb903dad80c36 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000140: Checksum does not match: 3072083029 vs expected 3072083029 W0720 00:44:04.244541 4358 catalog_manager.cc:3988] Tablet 27285c1e3c36466f9bce8450be7500d4 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000176: Checksum does not match: 2919875488 vs expected 2919875488 W0720 00:44:04.244616 4358 catalog_manager.cc:3988] Tablet 5758df4920ce4bbab0424048f13fea41 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000152: Checksum does not match: 1478762943 vs expected 1478762943 W0720 00:44:04.244680 4358 catalog_manager.cc:3988] Tablet 73ea05088cb24563bcf684d6199dd946 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000212: Checksum does not match: 3440600655 vs expected 3440600655 W0720 00:44:04.244757 4358 catalog_manager.cc:3988] Tablet ca6a2a19cbbe4648a8a76de1e43c405b (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000224: Checksum does not match: 1683009991 vs expected 1683009991 W0720 00:44:04.244822 4358 catalog_manager.cc:3988] Tablet c4890cfc794b469d8765701546219a19 (table test-workload [id=03430f12567b4e1ab531baa411322323]) has failed on TS 506b8430b3284890b468e7ef55b6ca79 (127.0.58.129:36375): Corruption: failed to init CFileReader for block 0000000000000200: Checksum does not match: 3565727781 vs expected 3565727781 I0720 00:44:04.277179 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 73ea05088cb24563bcf684d6199dd946 I0720 00:44:04.277314 234 log_verifier.cc:121] Checking tablet 7869d31b2ece4c38b8cf73ba177d2787 W0720 00:44:04.290926 5312 consensus_peers.cc:251] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.291400 5312 consensus_peers.cc:251] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.351119 234 log_verifier.cc:172] Verified matching terms for 105 ops in tablet 7869d31b2ece4c38b8cf73ba177d2787 I0720 00:44:04.351284 234 log_verifier.cc:121] Checking tablet 9ef65edd881743f691ebb903dad80c36 W0720 00:44:04.361094 5312 consensus_peers.cc:251] T 73ea05088cb24563bcf684d6199dd946 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.361533 5277 consensus_peers.cc:251] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.423290 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 9ef65edd881743f691ebb903dad80c36 I0720 00:44:04.423462 234 log_verifier.cc:121] Checking tablet c4890cfc794b469d8765701546219a19 I0720 00:44:04.497504 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet c4890cfc794b469d8765701546219a19 W0720 00:44:04.497561 5277 consensus_peers.cc:251] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.497694 234 log_verifier.cc:121] Checking tablet ca6a2a19cbbe4648a8a76de1e43c405b W0720 00:44:04.541821 5277 consensus_peers.cc:251] T 0a7ad36f80f44067ba0087a2768e26ac P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.572460 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet ca6a2a19cbbe4648a8a76de1e43c405b I0720 00:44:04.572729 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 5114 I0720 00:44:04.573551 5237 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:04.592056 5277 consensus_peers.cc:251] T 73ac7512c8584fe3ad887ffef570df8d P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.598275 5277 consensus_peers.cc:251] T 7869d31b2ece4c38b8cf73ba177d2787 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.615274 5312 consensus_peers.cc:251] T ca6a2a19cbbe4648a8a76de1e43c405b P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.624603 5312 consensus_peers.cc:251] T 9ef65edd881743f691ebb903dad80c36 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.627537 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 5114 I0720 00:44:04.638211 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 4534 I0720 00:44:04.639046 4652 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:04.753091 5277 consensus_peers.cc:251] T 5758df4920ce4bbab0424048f13fea41 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.789911 5312 consensus_peers.cc:251] T c4890cfc794b469d8765701546219a19 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:04.790436 5342 consensus_peers.cc:251] T 27285c1e3c36466f9bce8450be7500d4 P a29a463e4c2e4f61a400dde426c625e8 -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.790750 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 4534 I0720 00:44:04.813521 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 4658 W0720 00:44:04.813520 4348 connection.cc:655] server connection from 127.0.58.130:35167 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:44:04.814371 4776 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:04.970110 5277 consensus_peers.cc:251] T 19920b309dd84a01b2cc74ddbbd10854 P fc86e9a330584f00b998288875c0586f -> Peer e23cbbcbc8444eec834170f2d501c129 (127.0.58.132:46503): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:04.973204 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 4658 I0720 00:44:04.996346 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 4980 I0720 00:44:04.997248 5101 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:05.041417 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 4980 I0720 00:44:05.050618 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 4336 I0720 00:44:05.051450 4394 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:05.134572 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 4336 [ OK ] DiskError/DiskErrorITest.TestFailOnBootstrap/0 (8609 ms) [ RUN ] DiskError/DiskErrorITest.TestFailOnBootstrap/1 I0720 00:44:05.184644 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-2 --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:38255 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_tombstone_evicted_tablet_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:05.382165 5350 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:05.382369 5350 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:05.384966 5350 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:44:05.385006 5350 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:05.385042 5350 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:44:05.385082 5350 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:44:05.389336 5350 master_main.cc:98] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/wal --master_tombstone_evicted_tablet_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:38255 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:05.395556 5350 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 84016us I0720 00:44:05.396736 5350 fs_manager.cc:263] Metadata directory not provided I0720 00:44:05.396811 5350 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:05.397110 5350 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:05.397182 5350 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:05.399560 5350 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "ad906cc98588459aa9641d8f4ea26ed9" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.400105 5350 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "ad906cc98588459aa9641d8f4ea26ed9" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.400439 5350 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "ad906cc98588459aa9641d8f4ea26ed9" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.400756 5350 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "ad906cc98588459aa9641d8f4ea26ed9" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.409894 5350 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s I0720 00:44:05.411124 5350 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:05.411640 5350 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:05.414764 5350 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0720 00:44:05.414819 5350 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/wal uuid: "ad906cc98588459aa9641d8f4ea26ed9" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.414976 5350 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:05.481847 5350 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:05.494307 5350 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:38255 I0720 00:44:05.495393 5350 webserver.cc:374] Webserver started at http://127.0.58.190:37355/ using document root and password file I0720 00:44:05.496094 5350 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:44:05.497002 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 5350 I0720 00:44:05.497740 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:38255 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:05.517793 5411 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9: Bootstrap starting. I0720 00:44:05.524673 5411 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9: No blocks or log segments found. Creating new log. I0720 00:44:05.525573 5411 log.cc:525] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9: Log is configured to *not* fsync() on all Append() calls I0720 00:44:05.531486 5411 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9: No bootstrap required, opened a new log I0720 00:44:05.540560 5411 raft_consensus.cc:340] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } I0720 00:44:05.540851 5411 raft_consensus.cc:366] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:05.540948 5411 raft_consensus.cc:705] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad906cc98588459aa9641d8f4ea26ed9, State: Initialized, Role: FOLLOWER I0720 00:44:05.541505 5411 consensus_queue.cc:229] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } I0720 00:44:05.541730 5411 raft_consensus.cc:380] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:44:05.541805 5411 raft_consensus.cc:472] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:44:05.541935 5411 raft_consensus.cc:2890] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:05.544302 5411 raft_consensus.cc:494] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } I0720 00:44:05.544826 5411 leader_election.cc:310] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad906cc98588459aa9641d8f4ea26ed9; no voters: I0720 00:44:05.545666 5411 leader_election.cc:296] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:44:05.545861 5414 raft_consensus.cc:2644] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:05.546732 5414 raft_consensus.cc:667] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [term 1 LEADER]: Becoming Leader. State: Replica: ad906cc98588459aa9641d8f4ea26ed9, State: Running, Role: LEADER I0720 00:44:05.547184 5414 consensus_queue.cc:206] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } I0720 00:44:05.550318 5416 sys_catalog.cc:340] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad906cc98588459aa9641d8f4ea26ed9. Latest consensus state: current_term: 1 leader_uuid: "ad906cc98588459aa9641d8f4ea26ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } } I0720 00:44:05.550947 5411 sys_catalog.cc:340] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ad906cc98588459aa9641d8f4ea26ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } } I0720 00:44:05.551074 5416 sys_catalog.cc:343] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: This master's current role is: LEADER I0720 00:44:05.551304 5415 sys_catalog.cc:340] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad906cc98588459aa9641d8f4ea26ed9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad906cc98588459aa9641d8f4ea26ed9" member_type: VOTER } } I0720 00:44:05.551738 5415 sys_catalog.cc:343] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: This master's current role is: LEADER I0720 00:44:05.551880 5411 sys_catalog.cc:343] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: This master's current role is: LEADER I0720 00:44:05.553359 5411 sys_catalog.cc:428] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9 [sys.catalog]: configured and running, proceeding with master startup. I0720 00:44:05.554486 5421 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:44:05.564249 5421 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:44:05.619784 5421 catalog_manager.cc:959] Generated new certificate authority record I0720 00:44:05.621572 5421 catalog_manager.cc:1084] Loading token signing keys... I0720 00:44:05.638160 5421 catalog_manager.cc:4371] T 00000000000000000000000000000000 P ad906cc98588459aa9641d8f4ea26ed9: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:05.682147 5412 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:05.682392 5412 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:05.684505 5412 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:05.684597 5412 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:05.684643 5412 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:05.684720 5412 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:44:05.698263 5412 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:38255 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:05.705868 5412 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 84016us I0720 00:44:05.706589 5412 fs_manager.cc:263] Metadata directory not provided I0720 00:44:05.706665 5412 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:05.706959 5412 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:05.707032 5412 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:05.709316 5412 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.709869 5412 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.710248 5412 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.710580 5412 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.728698 5412 fs_manager.cc:503] Time spent creating directory manager: real 0.018s user 0.006s sys 0.011s I0720 00:44:05.730723 5412 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:05.731415 5412 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:05.735381 5412 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0720 00:44:05.735460 5412 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:05.735659 5412 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:05.794672 5412 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:05.799393 5412 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:05.799471 5412 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:05.867213 5412 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:34347 I0720 00:44:05.868477 5412 webserver.cc:374] Webserver started at http://127.0.58.129:42415/ using document root and password file I0720 00:44:05.869143 5412 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:44:05.872390 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 5412 I0720 00:44:05.873140 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:38255 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:05.893272 5544 heartbeater.cc:324] Connected to a master server at 127.0.58.190:38255 I0720 00:44:05.893481 5544 heartbeater.cc:405] Registering TS with master... I0720 00:44:05.893959 5544 heartbeater.cc:454] Master 127.0.58.190:38255 requested a full tablet report, sending... I0720 00:44:05.895604 5378 ts_manager.cc:161] Registered new tserver with Master: be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:05.897083 5378 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:52789 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:06.033725 5548 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:06.033954 5548 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:06.036039 5548 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:06.036130 5548 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:06.036176 5548 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:06.036257 5548 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:44:06.039943 5548 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:38255 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:06.047350 5548 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 84516us I0720 00:44:06.047992 5548 fs_manager.cc:263] Metadata directory not provided I0720 00:44:06.048065 5548 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:06.048352 5548 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:06.048430 5548 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:06.050685 5548 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "445c7e479f0246a5b26400b1cd4d8d27" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.051208 5548 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "445c7e479f0246a5b26400b1cd4d8d27" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.051551 5548 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "445c7e479f0246a5b26400b1cd4d8d27" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.051883 5548 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "445c7e479f0246a5b26400b1cd4d8d27" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.060135 5548 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s I0720 00:44:06.061345 5548 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:06.061836 5548 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:06.064797 5548 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:44:06.064865 5548 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/wal uuid: "445c7e479f0246a5b26400b1cd4d8d27" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.065016 5548 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:06.092772 5548 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:06.094358 5548 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:06.094417 5548 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:06.117475 5548 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:39833 I0720 00:44:06.118602 5548 webserver.cc:374] Webserver started at http://127.0.58.130:34857/ using document root and password file I0720 00:44:06.119235 5548 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:44:06.119657 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 5548 I0720 00:44:06.120434 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:38255 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:06.139382 5669 heartbeater.cc:324] Connected to a master server at 127.0.58.190:38255 I0720 00:44:06.139622 5669 heartbeater.cc:405] Registering TS with master... I0720 00:44:06.140172 5669 heartbeater.cc:454] Master 127.0.58.190:38255 requested a full tablet report, sending... I0720 00:44:06.141398 5378 ts_manager.cc:161] Registered new tserver with Master: 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833) I0720 00:44:06.142273 5378 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:36855 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:06.281927 5672 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:06.282184 5672 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:06.284312 5672 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:06.284402 5672 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:06.284445 5672 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:06.284533 5672 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:44:06.288300 5672 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:38255 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:06.295928 5672 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 84516us I0720 00:44:06.296576 5672 fs_manager.cc:263] Metadata directory not provided I0720 00:44:06.296644 5672 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:06.296927 5672 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:06.297001 5672 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:06.299301 5672 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "aedbe741f4014e40ae382c213ab0d151" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.299825 5672 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "aedbe741f4014e40ae382c213ab0d151" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.300177 5672 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "aedbe741f4014e40ae382c213ab0d151" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.300506 5672 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "aedbe741f4014e40ae382c213ab0d151" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.308421 5672 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.007s sys 0.003s I0720 00:44:06.309649 5672 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:06.310173 5672 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:06.313125 5672 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:44:06.313187 5672 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/wal uuid: "aedbe741f4014e40ae382c213ab0d151" format_stamp: "Formatted at 2019-07-20 00:44:06 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:06.313362 5672 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:06.341293 5672 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:06.342866 5672 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:06.342916 5672 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:06.365670 5672 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:46091 I0720 00:44:06.366772 5672 webserver.cc:374] Webserver started at http://127.0.58.131:40329/ using document root and password file I0720 00:44:06.367400 5672 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:44:06.374855 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 5672 I0720 00:44:06.380770 5793 heartbeater.cc:324] Connected to a master server at 127.0.58.190:38255 I0720 00:44:06.380973 5793 heartbeater.cc:405] Registering TS with master... I0720 00:44:06.381546 5793 heartbeater.cc:454] Master 127.0.58.190:38255 requested a full tablet report, sending... I0720 00:44:06.382691 5378 ts_manager.cc:161] Registered new tserver with Master: aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.383481 5378 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:46599 I0720 00:44:06.384594 234 external_mini_cluster.cc:628] 3 TS(s) registered with all masters I0720 00:44:06.394868 234 test_util.cc:212] Using random seed: 602286044 I0720 00:44:06.408432 5378 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37520: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s" } partition_schema { range_schema { columns { name: "key" } } } W0720 00:44:06.410133 5378 catalog_manager.cc:1583] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0720 00:44:06.439772 5730 tablet_service.cc:1022] Processing CreateTablet for tablet 6c0c6e70945746faa55c4964f07a5a86 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:06.441030 5606 tablet_service.cc:1022] Processing CreateTablet for tablet 6c0c6e70945746faa55c4964f07a5a86 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:06.441879 5472 tablet_service.cc:1022] Processing CreateTablet for tablet f0f53e18f4ec4eec823ef551f843b54e (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:06.441036 5476 tablet_service.cc:1022] Processing CreateTablet for tablet 43e3ab995f34471c92e887c28c2e2e0f (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:06.442484 5473 tablet_service.cc:1022] Processing CreateTablet for tablet c1e0c42e42524c22af51c8e4a457d743 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:06.446523 5605 tablet_service.cc:1022] Processing CreateTablet for tablet f50d69e061a145869a005de2b35d1acc (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:06.446715 5729 tablet_service.cc:1022] Processing CreateTablet for tablet f50d69e061a145869a005de2b35d1acc (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:06.442572 5477 tablet_service.cc:1022] Processing CreateTablet for tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:06.447968 5604 tablet_service.cc:1022] Processing CreateTablet for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:06.448022 5603 tablet_service.cc:1022] Processing CreateTablet for tablet d54fc0a6131248cc929ba6d4b0a17489 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:06.454149 5602 tablet_service.cc:1022] Processing CreateTablet for tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:06.454675 5728 tablet_service.cc:1022] Processing CreateTablet for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:06.455013 5474 tablet_service.cc:1022] Processing CreateTablet for tablet 736b3f88dd4a41fbb23ef072e4e28905 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:06.456348 5601 tablet_service.cc:1022] Processing CreateTablet for tablet 43e3ab995f34471c92e887c28c2e2e0f (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:06.455212 5478 tablet_service.cc:1022] Processing CreateTablet for tablet d54fc0a6131248cc929ba6d4b0a17489 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:06.455350 5481 tablet_service.cc:1022] Processing CreateTablet for tablet 6c0c6e70945746faa55c4964f07a5a86 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:06.455463 5479 tablet_service.cc:1022] Processing CreateTablet for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:06.455579 5480 tablet_service.cc:1022] Processing CreateTablet for tablet f50d69e061a145869a005de2b35d1acc (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:06.458657 5727 tablet_service.cc:1022] Processing CreateTablet for tablet d54fc0a6131248cc929ba6d4b0a17489 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:06.455675 5475 tablet_service.cc:1022] Processing CreateTablet for tablet dff7af31a11e4cf78f6efbbfa12762d3 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:06.462039 5600 tablet_service.cc:1022] Processing CreateTablet for tablet dff7af31a11e4cf78f6efbbfa12762d3 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:06.463420 5599 tablet_service.cc:1022] Processing CreateTablet for tablet 736b3f88dd4a41fbb23ef072e4e28905 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:06.464628 5726 tablet_service.cc:1022] Processing CreateTablet for tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:06.465101 5598 tablet_service.cc:1022] Processing CreateTablet for tablet c1e0c42e42524c22af51c8e4a457d743 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:06.467013 5597 tablet_service.cc:1022] Processing CreateTablet for tablet f0f53e18f4ec4eec823ef551f843b54e (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:06.472613 5725 tablet_service.cc:1022] Processing CreateTablet for tablet 43e3ab995f34471c92e887c28c2e2e0f (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:06.478703 5724 tablet_service.cc:1022] Processing CreateTablet for tablet dff7af31a11e4cf78f6efbbfa12762d3 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:06.482656 5723 tablet_service.cc:1022] Processing CreateTablet for tablet 736b3f88dd4a41fbb23ef072e4e28905 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:06.486917 5722 tablet_service.cc:1022] Processing CreateTablet for tablet c1e0c42e42524c22af51c8e4a457d743 (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:06.491818 5721 tablet_service.cc:1022] Processing CreateTablet for tablet f0f53e18f4ec4eec823ef551f843b54e (table=test-workload [id=34d3a1886d964668a2d9020148c19d10]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:06.527777 5817 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.530588 5818 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.533082 5818 tablet_bootstrap.cc:590] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.534497 5819 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.536152 5819 tablet_bootstrap.cc:590] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.537009 5819 log.cc:525] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: Log is configured to *not* fsync() on all Append() calls I0720 00:44:06.547106 5821 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.552458 5823 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.555287 5822 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.555747 5817 tablet_bootstrap.cc:590] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.556078 5825 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.558837 5821 tablet_bootstrap.cc:590] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.557076 5819 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.559206 5819 ts_tablet_manager.cc:1059] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s I0720 00:44:06.559962 5821 log.cc:525] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27: Log is configured to *not* fsync() on all Append() calls I0720 00:44:06.560488 5823 tablet_bootstrap.cc:590] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.561529 5822 tablet_bootstrap.cc:590] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.562541 5818 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.562713 5818 ts_tablet_manager.cc:1059] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s I0720 00:44:06.561651 5821 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.563096 5821 ts_tablet_manager.cc:1059] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s I0720 00:44:06.564157 5817 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.562057 5826 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.564298 5817 ts_tablet_manager.cc:1059] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s I0720 00:44:06.566715 5822 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.568244 5822 ts_tablet_manager.cc:1059] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s I0720 00:44:06.569792 5823 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.569787 5824 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.569954 5823 ts_tablet_manager.cc:1059] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s I0720 00:44:06.574203 5821 raft_consensus.cc:340] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.574690 5821 raft_consensus.cc:366] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.573778 5823 raft_consensus.cc:340] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.574124 5822 raft_consensus.cc:340] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.575083 5822 raft_consensus.cc:366] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.575178 5822 raft_consensus.cc:705] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.575407 5825 tablet_bootstrap.cc:590] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.574828 5821 raft_consensus.cc:705] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.575888 5821 consensus_queue.cc:229] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.574959 5823 raft_consensus.cc:366] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.576467 5826 tablet_bootstrap.cc:590] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.576472 5823 raft_consensus.cc:705] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.576895 5823 consensus_queue.cc:229] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.577039 5825 log.cc:525] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Log is configured to *not* fsync() on all Append() calls I0720 00:44:06.577183 5822 consensus_queue.cc:229] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.577473 5669 heartbeater.cc:446] Master 127.0.58.190:38255 was elected leader, sending a full tablet report... I0720 00:44:06.578471 5823 ts_tablet_manager.cc:1087] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s I0720 00:44:06.578065 5824 tablet_bootstrap.cc:590] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.578996 5823 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.581368 5819 raft_consensus.cc:340] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.581956 5825 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.582336 5825 ts_tablet_manager.cc:1059] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s I0720 00:44:06.582511 5819 raft_consensus.cc:366] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.582286 5818 raft_consensus.cc:340] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.582659 5818 raft_consensus.cc:366] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.582759 5818 raft_consensus.cc:705] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.581959 5826 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.583433 5826 ts_tablet_manager.cc:1059] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s I0720 00:44:06.583310 5818 consensus_queue.cc:229] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.581957 5817 raft_consensus.cc:340] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.583838 5817 raft_consensus.cc:366] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.583926 5817 raft_consensus.cc:705] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.583979 5824 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.584126 5817 consensus_queue.cc:229] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.584497 5819 raft_consensus.cc:705] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.584738 5793 heartbeater.cc:446] Master 127.0.58.190:38255 was elected leader, sending a full tablet report... I0720 00:44:06.585290 5821 ts_tablet_manager.cc:1087] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.015s user 0.005s sys 0.004s I0720 00:44:06.585346 5818 ts_tablet_manager.cc:1087] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s I0720 00:44:06.585350 5824 ts_tablet_manager.cc:1059] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s I0720 00:44:06.585479 5819 consensus_queue.cc:229] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.585566 5821 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.585716 5818 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.587188 5818 tablet_bootstrap.cc:590] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.587487 5821 tablet_bootstrap.cc:590] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.588865 5819 ts_tablet_manager.cc:1087] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s I0720 00:44:06.589237 5819 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.590392 5817 ts_tablet_manager.cc:1087] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s I0720 00:44:06.590662 5817 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.591038 5818 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.591809 5819 tablet_bootstrap.cc:590] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.592007 5817 tablet_bootstrap.cc:590] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.592072 5818 ts_tablet_manager.cc:1059] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s I0720 00:44:06.593070 5818 raft_consensus.cc:340] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.593484 5818 raft_consensus.cc:366] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.593874 5822 ts_tablet_manager.cc:1087] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.025s user 0.000s sys 0.003s I0720 00:44:06.594125 5818 raft_consensus.cc:705] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.594261 5822 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.594543 5819 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.595072 5823 tablet_bootstrap.cc:590] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.595136 5819 ts_tablet_manager.cc:1059] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s I0720 00:44:06.595845 5819 raft_consensus.cc:340] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.596537 5819 raft_consensus.cc:366] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.596627 5822 tablet_bootstrap.cc:590] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.596640 5819 raft_consensus.cc:705] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.594712 5818 consensus_queue.cc:229] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.596877 5821 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.597229 5821 ts_tablet_manager.cc:1059] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s I0720 00:44:06.598182 5823 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.598372 5823 ts_tablet_manager.cc:1059] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s I0720 00:44:06.598574 5821 raft_consensus.cc:340] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.598894 5821 raft_consensus.cc:366] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.599022 5821 raft_consensus.cc:705] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.599383 5818 ts_tablet_manager.cc:1087] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s I0720 00:44:06.599627 5818 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.599818 5823 raft_consensus.cc:340] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.600906 5823 raft_consensus.cc:366] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.600950 5818 tablet_bootstrap.cc:590] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.601001 5823 raft_consensus.cc:705] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.600353 5821 consensus_queue.cc:229] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.601899 5821 ts_tablet_manager.cc:1087] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s I0720 00:44:06.600832 5822 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.602224 5817 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.602252 5822 ts_tablet_manager.cc:1059] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s I0720 00:44:06.602417 5817 ts_tablet_manager.cc:1059] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s I0720 00:44:06.602815 5822 raft_consensus.cc:340] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.603013 5817 raft_consensus.cc:340] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.603422 5822 raft_consensus.cc:366] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.603010 5825 raft_consensus.cc:340] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.603469 5817 raft_consensus.cc:366] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.603523 5817 raft_consensus.cc:705] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.603535 5822 raft_consensus.cc:705] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.603538 5825 raft_consensus.cc:366] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.603603 5825 raft_consensus.cc:705] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.603132 5821 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.603744 5817 consensus_queue.cc:229] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.603293 5819 consensus_queue.cc:229] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.603881 5826 raft_consensus.cc:340] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.604411 5826 raft_consensus.cc:366] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.604496 5819 ts_tablet_manager.cc:1087] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s I0720 00:44:06.604507 5826 raft_consensus.cc:705] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.604758 5819 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.604938 5821 tablet_bootstrap.cc:590] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.604887 5826 consensus_queue.cc:229] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.605528 5824 raft_consensus.cc:340] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.605763 5824 raft_consensus.cc:366] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.605865 5824 raft_consensus.cc:705] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.603240 5823 consensus_queue.cc:229] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.606112 5819 tablet_bootstrap.cc:590] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.606124 5824 consensus_queue.cc:229] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.607453 5817 ts_tablet_manager.cc:1087] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s I0720 00:44:06.608307 5817 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.608469 5819 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.608650 5819 ts_tablet_manager.cc:1059] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s I0720 00:44:06.607450 5825 consensus_queue.cc:229] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.609421 5823 ts_tablet_manager.cc:1087] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s I0720 00:44:06.609475 5544 heartbeater.cc:446] Master 127.0.58.190:38255 was elected leader, sending a full tablet report... I0720 00:44:06.609684 5823 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.605108 5822 consensus_queue.cc:229] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.610430 5819 raft_consensus.cc:340] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.610816 5819 raft_consensus.cc:366] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.610903 5819 raft_consensus.cc:705] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.611163 5819 consensus_queue.cc:229] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.612433 5822 ts_tablet_manager.cc:1087] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s I0720 00:44:06.613185 5822 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.613977 5823 tablet_bootstrap.cc:590] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.615203 5825 ts_tablet_manager.cc:1087] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.033s user 0.009s sys 0.003s I0720 00:44:06.615695 5825 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.616386 5819 ts_tablet_manager.cc:1087] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s I0720 00:44:06.616662 5819 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151: Bootstrap starting. I0720 00:44:06.617183 5825 tablet_bootstrap.cc:590] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.617880 5821 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.618014 5819 tablet_bootstrap.cc:590] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.618068 5821 ts_tablet_manager.cc:1059] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s I0720 00:44:06.618618 5824 ts_tablet_manager.cc:1087] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.033s user 0.000s sys 0.008s I0720 00:44:06.611517 5817 tablet_bootstrap.cc:590] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151: No blocks or log segments found. Creating new log. I0720 00:44:06.618901 5824 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.619077 5825 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log W0720 00:44:06.619176 5794 tablet.cc:1799] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:06.619768 5821 raft_consensus.cc:340] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.611390 5818 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.620060 5818 ts_tablet_manager.cc:1059] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s I0720 00:44:06.620393 5824 tablet_bootstrap.cc:590] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. W0720 00:44:06.620607 5670 tablet.cc:1799] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:06.620663 5821 raft_consensus.cc:366] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.620712 5821 raft_consensus.cc:705] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.616369 5822 tablet_bootstrap.cc:590] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.621330 5826 ts_tablet_manager.cc:1087] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.037s user 0.003s sys 0.005s I0720 00:44:06.621596 5826 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. W0720 00:44:06.622068 5545 tablet.cc:1799] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:06.622171 5823 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.622282 5823 ts_tablet_manager.cc:1059] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s I0720 00:44:06.620411 5825 ts_tablet_manager.cc:1059] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s I0720 00:44:06.622843 5823 raft_consensus.cc:340] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.622905 5818 raft_consensus.cc:340] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.623030 5823 raft_consensus.cc:366] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.623085 5823 raft_consensus.cc:705] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.623114 5818 raft_consensus.cc:366] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.623206 5818 raft_consensus.cc:705] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.623323 5825 raft_consensus.cc:340] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.623282 5823 consensus_queue.cc:229] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.623492 5825 raft_consensus.cc:366] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.623420 5818 consensus_queue.cc:229] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.624130 5823 ts_tablet_manager.cc:1087] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:06.624408 5823 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27: Bootstrap starting. I0720 00:44:06.624414 5818 ts_tablet_manager.cc:1087] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0720 00:44:06.624534 5825 raft_consensus.cc:705] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.624922 5819 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.624837 5825 consensus_queue.cc:229] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.624467 5826 tablet_bootstrap.cc:590] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.625749 5823 tablet_bootstrap.cc:590] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27: No blocks or log segments found. Creating new log. I0720 00:44:06.625846 5822 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.626205 5822 ts_tablet_manager.cc:1059] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s I0720 00:44:06.626353 5819 ts_tablet_manager.cc:1059] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s I0720 00:44:06.627070 5819 raft_consensus.cc:340] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.628047 5823 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27: No bootstrap required, opened a new log I0720 00:44:06.628206 5823 ts_tablet_manager.cc:1059] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s I0720 00:44:06.628334 5822 raft_consensus.cc:340] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.628584 5825 ts_tablet_manager.cc:1087] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s I0720 00:44:06.628901 5825 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.629298 5822 raft_consensus.cc:366] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.629413 5819 raft_consensus.cc:366] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.629521 5822 raft_consensus.cc:705] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.629629 5819 raft_consensus.cc:705] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.622062 5821 consensus_queue.cc:229] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.629782 5826 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.629933 5826 ts_tablet_manager.cc:1059] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s I0720 00:44:06.630007 5824 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.630380 5825 tablet_bootstrap.cc:590] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.630479 5821 ts_tablet_manager.cc:1087] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s I0720 00:44:06.628823 5823 raft_consensus.cc:340] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.630731 5823 raft_consensus.cc:366] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.629230 5817 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151: No bootstrap required, opened a new log I0720 00:44:06.630722 5826 raft_consensus.cc:340] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.630445 5824 ts_tablet_manager.cc:1059] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s I0720 00:44:06.631052 5817 ts_tablet_manager.cc:1059] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s I0720 00:44:06.631572 5824 raft_consensus.cc:340] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.631716 5817 raft_consensus.cc:340] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.631959 5817 raft_consensus.cc:366] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.632031 5817 raft_consensus.cc:705] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Initialized, Role: FOLLOWER I0720 00:44:06.632313 5824 raft_consensus.cc:366] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.632206 5817 consensus_queue.cc:229] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.632411 5824 raft_consensus.cc:705] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.632565 5825 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.630971 5826 raft_consensus.cc:366] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.632704 5826 raft_consensus.cc:705] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.632704 5824 consensus_queue.cc:229] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.632871 5826 consensus_queue.cc:229] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.633111 5817 ts_tablet_manager.cc:1087] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:06.633255 5825 ts_tablet_manager.cc:1059] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I0720 00:44:06.633296 5823 raft_consensus.cc:705] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Initialized, Role: FOLLOWER I0720 00:44:06.633774 5819 consensus_queue.cc:229] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.633455 5826 ts_tablet_manager.cc:1087] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s I0720 00:44:06.634210 5823 consensus_queue.cc:229] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.634367 5826 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.633483 5822 consensus_queue.cc:229] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.634946 5819 ts_tablet_manager.cc:1087] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s I0720 00:44:06.635681 5826 tablet_bootstrap.cc:590] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.636677 5824 ts_tablet_manager.cc:1087] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s I0720 00:44:06.636940 5824 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.637529 5822 ts_tablet_manager.cc:1087] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s I0720 00:44:06.634016 5825 raft_consensus.cc:340] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.638005 5825 raft_consensus.cc:366] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.637820 5823 ts_tablet_manager.cc:1087] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s I0720 00:44:06.638211 5824 tablet_bootstrap.cc:590] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.638216 5825 raft_consensus.cc:705] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.639220 5825 consensus_queue.cc:229] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.640254 5826 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.640367 5826 ts_tablet_manager.cc:1059] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s I0720 00:44:06.640405 5825 ts_tablet_manager.cc:1087] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s I0720 00:44:06.640774 5825 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. I0720 00:44:06.641295 5826 raft_consensus.cc:340] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.641564 5826 raft_consensus.cc:366] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.641625 5824 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.641682 5826 raft_consensus.cc:705] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.641794 5824 ts_tablet_manager.cc:1059] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s I0720 00:44:06.641922 5826 consensus_queue.cc:229] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.642261 5825 tablet_bootstrap.cc:590] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: No blocks or log segments found. Creating new log. I0720 00:44:06.642580 5826 ts_tablet_manager.cc:1087] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:06.642575 5824 raft_consensus.cc:340] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.642781 5824 raft_consensus.cc:366] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.642876 5824 raft_consensus.cc:705] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.643111 5824 consensus_queue.cc:229] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.643733 5824 ts_tablet_manager.cc:1087] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:06.643836 5825 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: No bootstrap required, opened a new log I0720 00:44:06.643937 5825 ts_tablet_manager.cc:1059] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s I0720 00:44:06.644465 5825 raft_consensus.cc:340] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.644595 5825 raft_consensus.cc:366] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:06.644641 5825 raft_consensus.cc:705] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Initialized, Role: FOLLOWER I0720 00:44:06.644774 5825 consensus_queue.cc:229] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.645102 5825 ts_tablet_manager.cc:1087] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0720 00:44:06.655920 5837 raft_consensus.cc:472] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.656137 5837 raft_consensus.cc:494] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.657531 5837 leader_election.cc:296] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:06.663815 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f0f53e18f4ec4eec823ef551f843b54e" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.664177 5501 raft_consensus.cc:2350] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.664645 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f0f53e18f4ec4eec823ef551f843b54e" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:06.664748 5682 leader_election.cc:310] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.664939 5626 raft_consensus.cc:2350] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.665158 5837 raft_consensus.cc:2644] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.665261 5837 raft_consensus.cc:472] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.665340 5837 raft_consensus.cc:2890] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.667366 5837 raft_consensus.cc:494] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.667838 5837 leader_election.cc:296] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Requested vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:06.668233 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f0f53e18f4ec4eec823ef551f843b54e" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:06.668264 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f0f53e18f4ec4eec823ef551f843b54e" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.668378 5626 raft_consensus.cc:2890] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.668390 5501 raft_consensus.cc:2890] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.669986 5501 raft_consensus.cc:2350] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.670056 5626 raft_consensus.cc:2350] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.670341 5682 leader_election.cc:310] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.670622 5837 raft_consensus.cc:2644] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.670734 5837 raft_consensus.cc:667] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Becoming Leader. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Running, Role: LEADER I0720 00:44:06.671090 5837 consensus_queue.cc:206] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.674599 5378 catalog_manager.cc:4067] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 reported cstate change: term changed from 0 to 1, leader changed from to aedbe741f4014e40ae382c213ab0d151 (127.0.58.131). New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:06.687960 5832 raft_consensus.cc:472] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.688107 5832 raft_consensus.cc:494] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.689301 5832 leader_election.cc:296] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.692502 5847 raft_consensus.cc:472] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.692697 5847 raft_consensus.cc:494] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.694206 5847 leader_election.cc:296] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.703119 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:06.703160 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6c0c6e70945746faa55c4964f07a5a86" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.703361 5626 raft_consensus.cc:2350] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 0. I0720 00:44:06.703378 5501 raft_consensus.cc:2350] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.703768 5435 leader_election.cc:310] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.703521 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:06.703984 5750 raft_consensus.cc:2350] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 0. I0720 00:44:06.704133 5847 raft_consensus.cc:2644] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.704224 5847 raft_consensus.cc:472] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.704298 5847 raft_consensus.cc:2890] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.704236 5558 leader_election.cc:310] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.704583 5832 raft_consensus.cc:2644] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.704658 5832 raft_consensus.cc:472] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.704718 5832 raft_consensus.cc:2890] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.705308 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6c0c6e70945746faa55c4964f07a5a86" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:06.705468 5750 raft_consensus.cc:2350] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.706357 5847 raft_consensus.cc:494] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.706614 5832 raft_consensus.cc:494] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.706807 5847 leader_election.cc:296] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 election: Requested vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.707106 5832 leader_election.cc:296] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Requested vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.707211 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:06.707339 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:06.707391 5626 raft_consensus.cc:2890] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.707499 5750 raft_consensus.cc:2890] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.707656 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6c0c6e70945746faa55c4964f07a5a86" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.707826 5501 raft_consensus.cc:2890] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.708730 5626 raft_consensus.cc:2350] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 1. I0720 00:44:06.709061 5435 leader_election.cc:310] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.709300 5847 raft_consensus.cc:2644] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.709415 5847 raft_consensus.cc:667] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 LEADER]: Becoming Leader. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Running, Role: LEADER I0720 00:44:06.709662 5750 raft_consensus.cc:2350] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 1. I0720 00:44:06.709728 5501 raft_consensus.cc:2350] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.709801 5847 consensus_queue.cc:206] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.710105 5558 leader_election.cc:310] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.710355 5832 raft_consensus.cc:2644] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.710340 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "6c0c6e70945746faa55c4964f07a5a86" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:06.710528 5750 raft_consensus.cc:2890] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.711484 5837 raft_consensus.cc:472] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.711638 5837 raft_consensus.cc:494] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.712711 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "dff7af31a11e4cf78f6efbbfa12762d3" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.712711 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "dff7af31a11e4cf78f6efbbfa12762d3" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:06.712869 5501 raft_consensus.cc:2350] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.712882 5626 raft_consensus.cc:2350] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.713279 5682 leader_election.cc:310] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.713729 5836 raft_consensus.cc:2644] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.713802 5836 raft_consensus.cc:472] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.713830 5832 raft_consensus.cc:667] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Becoming Leader. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Running, Role: LEADER I0720 00:44:06.713929 5836 raft_consensus.cc:2890] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.714241 5832 consensus_queue.cc:206] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.714896 5837 leader_election.cc:296] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:06.715054 5378 catalog_manager.cc:4067] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e reported cstate change: term changed from 0 to 1, leader changed from to be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } I0720 00:44:06.716084 5836 raft_consensus.cc:494] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.716678 5836 leader_election.cc:296] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Requested vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:06.717062 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "dff7af31a11e4cf78f6efbbfa12762d3" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:06.717195 5626 raft_consensus.cc:2890] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.717226 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "dff7af31a11e4cf78f6efbbfa12762d3" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.717367 5501 raft_consensus.cc:2890] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.718150 5375 catalog_manager.cc:4067] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: term changed from 0 to 1, leader changed from to 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } I0720 00:44:06.719317 5501 raft_consensus.cc:2350] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.719542 5626 raft_consensus.cc:2350] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.719692 5682 leader_election.cc:310] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.720041 5836 raft_consensus.cc:2644] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.720167 5836 raft_consensus.cc:667] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Becoming Leader. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Running, Role: LEADER I0720 00:44:06.720445 5836 consensus_queue.cc:206] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.720885 5750 raft_consensus.cc:2350] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.722630 5375 catalog_manager.cc:4067] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 reported cstate change: term changed from 0 to 1, leader changed from to aedbe741f4014e40ae382c213ab0d151 (127.0.58.131). New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:06.751557 5836 raft_consensus.cc:472] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.751708 5836 raft_consensus.cc:494] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.752354 5836 leader_election.cc:296] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:06.753001 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "43e3ab995f34471c92e887c28c2e2e0f" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:06.753001 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "43e3ab995f34471c92e887c28c2e2e0f" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.753224 5501 raft_consensus.cc:2350] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.753224 5626 raft_consensus.cc:2350] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.753643 5682 leader_election.cc:310] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.753952 5836 raft_consensus.cc:2644] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.754029 5836 raft_consensus.cc:472] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.754101 5836 raft_consensus.cc:2890] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.755174 5837 raft_consensus.cc:472] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.755297 5837 raft_consensus.cc:494] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.755831 5837 leader_election.cc:296] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833) I0720 00:44:06.756012 5836 raft_consensus.cc:494] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.756230 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "26b0b6bdff8a4bc7be9e3eeff65c3b9a" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:06.756232 5501 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "26b0b6bdff8a4bc7be9e3eeff65c3b9a" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.756429 5501 raft_consensus.cc:2350] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.756428 5626 raft_consensus.cc:2350] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 0. I0720 00:44:06.756695 5682 leader_election.cc:310] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.756786 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "43e3ab995f34471c92e887c28c2e2e0f" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:06.756944 5837 raft_consensus.cc:2644] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.756948 5626 raft_consensus.cc:2890] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.756891 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "43e3ab995f34471c92e887c28c2e2e0f" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.757005 5837 raft_consensus.cc:472] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.757056 5837 raft_consensus.cc:2890] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.757043 5500 raft_consensus.cc:2890] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.758481 5847 raft_consensus.cc:472] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.758481 5626 raft_consensus.cc:2350] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.758592 5847 raft_consensus.cc:494] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.758808 5684 leader_election.cc:310] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: I0720 00:44:06.758924 5837 raft_consensus.cc:494] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.759263 5500 raft_consensus.cc:2350] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.759529 5837 leader_election.cc:296] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Requested vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833) I0720 00:44:06.759657 5837 raft_consensus.cc:2644] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.759539 5625 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:06.759737 5837 raft_consensus.cc:667] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Becoming Leader. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Running, Role: LEADER I0720 00:44:06.759759 5625 raft_consensus.cc:2350] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 0. I0720 00:44:06.759938 5837 consensus_queue.cc:206] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.760191 5435 leader_election.cc:310] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.760341 5836 leader_election.cc:296] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Requested vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:06.760444 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "26b0b6bdff8a4bc7be9e3eeff65c3b9a" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.760499 5849 raft_consensus.cc:2644] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.760685 5500 raft_consensus.cc:2890] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.761955 5849 raft_consensus.cc:472] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.762068 5849 raft_consensus.cc:2890] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.763207 5500 raft_consensus.cc:2350] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.763242 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:06.763466 5750 raft_consensus.cc:2350] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 0. I0720 00:44:06.763612 5682 leader_election.cc:310] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.763334 5375 catalog_manager.cc:4067] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 reported cstate change: term changed from 0 to 1, leader changed from to aedbe741f4014e40ae382c213ab0d151 (127.0.58.131). New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:06.763993 5873 raft_consensus.cc:2644] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.764010 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "26b0b6bdff8a4bc7be9e3eeff65c3b9a" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:06.764217 5626 raft_consensus.cc:2890] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.764174 5873 raft_consensus.cc:667] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Becoming Leader. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Running, Role: LEADER I0720 00:44:06.764206 5849 raft_consensus.cc:494] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.764422 5873 consensus_queue.cc:206] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.764760 5849 leader_election.cc:296] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 election: Requested vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.765395 5847 leader_election.cc:296] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.765419 5625 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:06.765537 5625 raft_consensus.cc:2890] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.765949 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:06.766104 5750 raft_consensus.cc:2890] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.767516 5625 raft_consensus.cc:2350] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 1. I0720 00:44:06.767892 5435 leader_election.cc:310] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.768087 5750 raft_consensus.cc:2350] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be9af1eb9bb14a7e8ef0cf71154d5e6e in term 1. I0720 00:44:06.768213 5847 raft_consensus.cc:2644] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.768729 5847 raft_consensus.cc:667] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 LEADER]: Becoming Leader. State: Replica: be9af1eb9bb14a7e8ef0cf71154d5e6e, State: Running, Role: LEADER I0720 00:44:06.769037 5626 raft_consensus.cc:2350] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:06.768976 5847 consensus_queue.cc:206] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.769866 5375 catalog_manager.cc:4067] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 reported cstate change: term changed from 0 to 1, leader changed from to aedbe741f4014e40ae382c213ab0d151 (127.0.58.131). New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:06.771183 5378 catalog_manager.cc:4067] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e reported cstate change: term changed from 0 to 1, leader changed from to be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129). New cstate: current_term: 1 leader_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } I0720 00:44:06.772845 5832 raft_consensus.cc:472] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.772969 5832 raft_consensus.cc:494] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.773573 5832 leader_election.cc:296] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.773967 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d54fc0a6131248cc929ba6d4b0a17489" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.773989 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d54fc0a6131248cc929ba6d4b0a17489" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:06.774152 5500 raft_consensus.cc:2350] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.774181 5750 raft_consensus.cc:2350] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.774485 5558 leader_election.cc:310] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.774755 5832 raft_consensus.cc:2644] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.774806 5832 raft_consensus.cc:472] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.774857 5832 raft_consensus.cc:2890] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.776319 5832 raft_consensus.cc:494] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.776769 5832 leader_election.cc:296] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Requested vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.777140 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d54fc0a6131248cc929ba6d4b0a17489" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.777137 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d54fc0a6131248cc929ba6d4b0a17489" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:06.777289 5500 raft_consensus.cc:2890] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.777325 5750 raft_consensus.cc:2890] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.779350 5750 raft_consensus.cc:2350] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.779361 5500 raft_consensus.cc:2350] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.779698 5559 leader_election.cc:310] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: I0720 00:44:06.779965 5832 raft_consensus.cc:2644] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.780079 5832 raft_consensus.cc:667] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Becoming Leader. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Running, Role: LEADER I0720 00:44:06.780275 5832 consensus_queue.cc:206] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.782178 5378 catalog_manager.cc:4067] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: term changed from 0 to 1, leader changed from to 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } I0720 00:44:06.809825 5832 raft_consensus.cc:472] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.809962 5832 raft_consensus.cc:494] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.810606 5832 leader_election.cc:296] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.811184 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "736b3f88dd4a41fbb23ef072e4e28905" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.811208 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "736b3f88dd4a41fbb23ef072e4e28905" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:06.811406 5750 raft_consensus.cc:2350] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.811399 5500 raft_consensus.cc:2350] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.811749 5559 leader_election.cc:310] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: I0720 00:44:06.812014 5832 raft_consensus.cc:2644] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.812085 5832 raft_consensus.cc:472] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.812144 5832 raft_consensus.cc:2890] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.813884 5832 raft_consensus.cc:494] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.814347 5832 leader_election.cc:296] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Requested vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.814779 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "736b3f88dd4a41fbb23ef072e4e28905" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.814807 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "736b3f88dd4a41fbb23ef072e4e28905" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:06.814951 5750 raft_consensus.cc:2890] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.814951 5500 raft_consensus.cc:2890] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.816948 5500 raft_consensus.cc:2350] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.816965 5750 raft_consensus.cc:2350] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.817302 5559 leader_election.cc:310] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: I0720 00:44:06.817533 5832 raft_consensus.cc:2644] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.817633 5832 raft_consensus.cc:667] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Becoming Leader. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Running, Role: LEADER I0720 00:44:06.817822 5832 consensus_queue.cc:206] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.820106 5378 catalog_manager.cc:4067] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: term changed from 0 to 1, leader changed from to 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } I0720 00:44:06.862594 5832 raft_consensus.cc:472] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:06.862771 5832 raft_consensus.cc:494] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.863325 5832 leader_election.cc:296] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.863956 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c1e0c42e42524c22af51c8e4a457d743" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:06.864142 5500 raft_consensus.cc:2350] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.864576 5558 leader_election.cc:310] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.864877 5832 raft_consensus.cc:2644] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:06.864933 5832 raft_consensus.cc:472] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:06.865000 5832 raft_consensus.cc:2890] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.866590 5832 raft_consensus.cc:494] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.867486 5750 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c1e0c42e42524c22af51c8e4a457d743" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:06.867610 5500 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c1e0c42e42524c22af51c8e4a457d743" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:06.867774 5750 raft_consensus.cc:2350] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 0. I0720 00:44:06.867794 5500 raft_consensus.cc:2890] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.867905 5832 leader_election.cc:296] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Requested vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:06.868569 5749 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c1e0c42e42524c22af51c8e4a457d743" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:06.868722 5749 raft_consensus.cc:2890] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:06.870018 5500 raft_consensus.cc:2350] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.870450 5558 leader_election.cc:310] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e; no voters: I0720 00:44:06.870731 5832 raft_consensus.cc:2644] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:06.870784 5749 raft_consensus.cc:2350] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. I0720 00:44:06.870856 5832 raft_consensus.cc:667] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Becoming Leader. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Running, Role: LEADER I0720 00:44:06.871057 5832 consensus_queue.cc:206] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:06.873901 5378 catalog_manager.cc:4067] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: term changed from 0 to 1, leader changed from to 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } I0720 00:44:06.964057 5500 raft_consensus.cc:1184] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:06.965173 5873 consensus_queue.cc:984] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:06.970235 5501 raft_consensus.cc:1184] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:06.970906 5873 consensus_queue.cc:984] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:06.982067 5626 raft_consensus.cc:1184] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:06.983070 5873 consensus_queue.cc:984] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:06.986371 5749 raft_consensus.cc:1184] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer be9af1eb9bb14a7e8ef0cf71154d5e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:06.987596 5847 consensus_queue.cc:984] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:06.992559 5750 raft_consensus.cc:1184] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:06.993317 5869 consensus_queue.cc:984] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:06.999297 5748 raft_consensus.cc:1184] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.010850 5748 raft_consensus.cc:1184] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.011909 5869 consensus_queue.cc:984] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.011909 5909 consensus_queue.cc:984] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:44:07.021780 5895 mvcc.cc:203] Tried to move safe_time back from 6404437798742478848 to 6404437797932630016. Current Snapshot: MvccSnapshot[committed={T|T < 6404437798742478848 or (T in {6404437798742478848})}] I0720 00:44:07.024938 5625 raft_consensus.cc:1184] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.026247 5625 raft_consensus.cc:1184] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.026568 5748 raft_consensus.cc:1184] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.026675 5837 consensus_queue.cc:984] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.027070 5836 consensus_queue.cc:984] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0720 00:44:07.028488 5909 consensus_queue.cc:984] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.023159 5903 mvcc.cc:203] Tried to move safe_time back from 6404437798742478848 to 6404437797932630016. Current Snapshot: MvccSnapshot[committed={T|T < 6404437798742478848}] I0720 00:44:07.113869 5626 raft_consensus.cc:1184] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer be9af1eb9bb14a7e8ef0cf71154d5e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.114107 5625 raft_consensus.cc:1184] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer be9af1eb9bb14a7e8ef0cf71154d5e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.142467 5745 raft_consensus.cc:1184] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer be9af1eb9bb14a7e8ef0cf71154d5e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.149580 5500 raft_consensus.cc:1184] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.150928 5849 consensus_queue.cc:984] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:44:07.151497 5847 consensus_queue.cc:984] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s I0720 00:44:07.152571 5497 raft_consensus.cc:1184] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:44:07.152585 5498 raft_consensus.cc:1184] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.153271 5847 consensus_queue.cc:984] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:44:07.154284 5496 raft_consensus.cc:1184] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.154824 5499 raft_consensus.cc:1184] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.157814 5495 raft_consensus.cc:1184] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:07.158260 5626 raft_consensus.cc:1184] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I0720 00:44:07.161120 5902 consensus_queue.cc:984] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.162459 5832 consensus_queue.cc:984] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.163682 5832 consensus_queue.cc:984] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.166625 5832 consensus_queue.cc:984] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0720 00:44:07.177639 5682 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:34347, user_credentials={real_user=slave}} blocked reactor thread for 63325us I0720 00:44:07.163708 5923 consensus_queue.cc:984] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s I0720 00:44:07.195449 5923 consensus_queue.cc:984] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.216166 5922 consensus_queue.cc:984] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:07.217479 5898 mvcc.cc:203] Tried to move safe_time back from 6404437798898266112 to 6404437798013652992. Current Snapshot: MvccSnapshot[committed={T|T < 6404437798898266112}] W0720 00:44:07.272589 5435 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:39833, user_credentials={real_user=slave}} blocked reactor thread for 80736.7us W0720 00:44:07.378118 5943 log.cc:669] Time spent T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s I0720 00:44:07.622956 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000288 I0720 00:44:07.624105 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000291 I0720 00:44:07.625659 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000281 I0720 00:44:07.711581 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.085s user 0.023s sys 0.004s Metrics: {"bytes_written":6295,"cfile_init":1,"compiler_manager_pool.queue_time_us":2768,"data dirs.queue_time_us":3665,"data dirs.run_cpu_time_us":198,"data dirs.run_wall_time_us":2504,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"rows_written":64,"spinlock_wait_cycles":273792,"thread_start_us":10024,"threads_started":2} I0720 00:44:07.713228 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 738 bytes on disk I0720 00:44:07.713934 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:07.715760 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.091s user 0.015s sys 0.018s Metrics: {"bytes_written":6408,"cfile_init":1,"compiler_manager_pool.queue_time_us":2146,"data dirs.queue_time_us":10206,"data dirs.run_cpu_time_us":160,"data dirs.run_wall_time_us":1596,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5218,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":67,"thread_start_us":12012,"threads_started":2} I0720 00:44:07.717316 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 750 bytes on disk I0720 00:44:07.717408 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000305 I0720 00:44:07.717973 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:07.718611 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000317 I0720 00:44:07.734215 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.015s user 0.015s sys 0.000s Metrics: {"bytes_written":6500,"cfile_init":1,"data dirs.queue_time_us":339,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1240,"lbm_writes_lt_1ms":23,"rows_written":72,"thread_start_us":428,"threads_started":1} I0720 00:44:07.735229 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000317 I0720 00:44:07.783928 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.049s user 0.011s sys 0.002s Metrics: {"bytes_written":6480,"cfile_init":1,"data dirs.queue_time_us":382,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"rows_written":71,"thread_start_us":459,"threads_started":1} I0720 00:44:07.784938 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 816 bytes on disk I0720 00:44:07.785445 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:07.786219 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 798 bytes on disk I0720 00:44:07.786756 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:07.787410 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000329 I0720 00:44:07.793715 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.076s user 0.017s sys 0.013s Metrics: {"bytes_written":6477,"cfile_init":1,"data dirs.queue_time_us":38076,"data dirs.run_cpu_time_us":178,"data dirs.run_wall_time_us":8107,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701,"lbm_writes_lt_1ms":23,"rows_written":71,"thread_start_us":37938,"threads_started":1} I0720 00:44:07.794775 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 789 bytes on disk I0720 00:44:07.795389 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4} I0720 00:44:07.796043 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000326 I0720 00:44:07.810029 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.014s user 0.007s sys 0.004s Metrics: {"bytes_written":7237,"cfile_init":1,"data dirs.queue_time_us":364,"data dirs.run_cpu_time_us":138,"data dirs.run_wall_time_us":497,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"rows_written":99,"thread_start_us":233,"threads_started":1} I0720 00:44:07.811033 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 1059 bytes on disk I0720 00:44:07.813686 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2234,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0720 00:44:07.815501 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000330 I0720 00:44:07.831256 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":6480,"cfile_init":1,"data dirs.queue_time_us":1686,"data dirs.run_cpu_time_us":143,"data dirs.run_wall_time_us":1047,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"rows_written":71} I0720 00:44:07.832278 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 798 bytes on disk I0720 00:44:07.832765 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:07.833426 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000333 I0720 00:44:07.840318 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.053s user 0.011s sys 0.001s Metrics: {"bytes_written":7039,"cfile_init":1,"data dirs.queue_time_us":2592,"data dirs.run_cpu_time_us":106,"data dirs.run_wall_time_us":10007,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"rows_written":91} I0720 00:44:07.847424 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6704,"cfile_init":1,"data dirs.queue_time_us":63,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":526,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":624,"lbm_writes_lt_1ms":23,"rows_written":79} I0720 00:44:07.848400 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000337 I0720 00:44:07.864506 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.016s user 0.009s sys 0.006s Metrics: {"bytes_written":7429,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":144,"data dirs.run_wall_time_us":478,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"rows_written":105} I0720 00:44:07.865615 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000340 I0720 00:44:07.876344 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000352 I0720 00:44:07.888696 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.023s user 0.009s sys 0.005s Metrics: {"bytes_written":7242,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":136,"data dirs.run_wall_time_us":11495,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"rows_written":98} I0720 00:44:07.889780 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000346 I0720 00:44:07.973104 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.095s user 0.014s sys 0.000s Metrics: {"bytes_written":7503,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":142,"data dirs.run_wall_time_us":573,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"rows_written":110} I0720 00:44:07.973865 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.350s user 0.015s sys 0.015s Metrics: {"bytes_written":6876,"cfile_init":1,"compiler_manager_pool.queue_time_us":4509,"data dirs.queue_time_us":476,"data dirs.run_cpu_time_us":160,"data dirs.run_wall_time_us":890,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"rows_written":86,"thread_start_us":68975,"threads_started":2} I0720 00:44:07.976977 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 927 bytes on disk I0720 00:44:07.977715 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4} I0720 00:44:07.978852 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 1150 bytes on disk I0720 00:44:07.979382 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:07.980109 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 974 bytes on disk I0720 00:44:07.980551 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:07.981214 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000380 I0720 00:44:07.982625 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000387 I0720 00:44:08.014019 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.122s user 0.011s sys 0.002s Metrics: {"bytes_written":7641,"cfile_init":1,"data dirs.queue_time_us":4207,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":609,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"rows_written":114} I0720 00:44:08.032430 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 1192 bytes on disk I0720 00:44:08.032913 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4} I0720 00:44:08.033478 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000386 I0720 00:44:08.044571 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.011s user 0.008s sys 0.000s Metrics: {"bytes_written":7329,"cfile_init":1,"data dirs.queue_time_us":64,"data dirs.run_cpu_time_us":155,"data dirs.run_wall_time_us":539,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":468,"lbm_writes_lt_1ms":23,"rows_written":103,"spinlock_wait_cycles":6400} I0720 00:44:08.045661 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000389 I0720 00:44:08.045737 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.064s user 0.005s sys 0.009s Metrics: {"bytes_written":7430,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":160,"data dirs.run_wall_time_us":516,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":668,"lbm_writes_lt_1ms":23,"rows_written":105} I0720 00:44:08.063030 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000400 I0720 00:44:08.067523 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.022s user 0.010s sys 0.005s Metrics: {"bytes_written":7846,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":120,"data dirs.run_wall_time_us":467,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5054,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":122} I0720 00:44:08.068624 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 1266 bytes on disk I0720 00:44:08.069180 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:08.069957 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 1135 bytes on disk I0720 00:44:08.070448 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:08.071080 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 1107 bytes on disk I0720 00:44:08.071602 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:08.072288 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 1050 bytes on disk I0720 00:44:08.072840 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:08.073892 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 873 bytes on disk I0720 00:44:08.082533 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4} W0720 00:44:08.101405 5684 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:39833, user_credentials={real_user=slave}} blocked reactor thread for 51038.1us I0720 00:44:08.117089 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.054s user 0.008s sys 0.006s Metrics: {"bytes_written":7496,"cfile_init":1,"data dirs.queue_time_us":707,"data dirs.run_cpu_time_us":141,"data dirs.run_wall_time_us":762,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"rows_written":109} W0720 00:44:08.174306 5684 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:39833, user_credentials={real_user=slave}} blocked reactor thread for 63541.2us I0720 00:44:08.207003 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000438 I0720 00:44:08.280437 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.296s user 0.011s sys 0.003s Metrics: {"bytes_written":7041,"cfile_init":1,"data dirs.queue_time_us":55,"data dirs.run_cpu_time_us":130,"data dirs.run_wall_time_us":476,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"rows_written":91} I0720 00:44:08.386839 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000499 I0720 00:44:08.431334 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.224s user 0.011s sys 0.006s Metrics: {"bytes_written":8099,"cfile_init":1,"data dirs.queue_time_us":14712,"data dirs.run_cpu_time_us":184,"data dirs.run_wall_time_us":898,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":734,"lbm_writes_lt_1ms":23,"rows_written":133} I0720 00:44:08.451145 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000499 I0720 00:44:08.612512 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.159s user 0.014s sys 0.003s Metrics: {"bytes_written":8895,"cfile_init":1,"data dirs.queue_time_us":5836,"data dirs.run_cpu_time_us":124,"data dirs.run_wall_time_us":1893,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"rows_written":163} I0720 00:44:08.613718 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 1678 bytes on disk I0720 00:44:08.614274 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4} I0720 00:44:08.615011 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 1389 bytes on disk I0720 00:44:08.615510 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:08.616191 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000549 I0720 00:44:08.647713 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.031s user 0.013s sys 0.002s Metrics: {"bytes_written":9502,"cfile_init":1,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":3017,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"rows_written":185,"thread_start_us":3272,"threads_started":1} I0720 00:44:08.651006 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 1903 bytes on disk I0720 00:44:08.651609 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:08.652380 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 1173 bytes on disk I0720 00:44:08.652922 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:08.653651 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 1133 bytes on disk I0720 00:44:08.658181 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:08.670308 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.283s user 0.012s sys 0.014s Metrics: {"bytes_written":8888,"cfile_init":1,"data dirs.queue_time_us":56918,"data dirs.run_cpu_time_us":179,"data dirs.run_wall_time_us":2486,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":673,"lbm_writes_lt_1ms":23,"rows_written":163,"thread_start_us":56793,"threads_started":1} I0720 00:44:08.671334 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 1674 bytes on disk I0720 00:44:08.671893 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4} I0720 00:44:08.672549 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000576 I0720 00:44:08.836259 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000312 I0720 00:44:08.909775 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000332 I0720 00:44:08.947135 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.037s user 0.013s sys 0.006s Metrics: {"bytes_written":8496,"cfile_init":1,"data dirs.queue_time_us":647,"data dirs.run_cpu_time_us":111,"data dirs.run_wall_time_us":463,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"rows_written":148} I0720 00:44:08.951670 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.981455 I0720 00:44:08.978866 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.291s user 0.053s sys 0.029s Metrics: {"bytes_written":9601,"cfile_init":1,"data dirs.queue_time_us":138160,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":3292,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"rows_written":191,"thread_start_us":138024,"threads_started":1} I0720 00:44:08.979984 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000660 I0720 00:44:09.003329 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.051s user 0.022s sys 0.004s Metrics: {"bytes_written":10201,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4135,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12223,"cfile_init":6,"data dirs.queue_time_us":4855,"data dirs.run_cpu_time_us":548,"data dirs.run_wall_time_us":3159,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":969,"lbm_reads_lt_1ms":50,"lbm_write_time_us":4039,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"rows_written":215,"thread_start_us":3597,"threads_started":1} I0720 00:44:09.004792 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 2169 bytes on disk I0720 00:44:09.005746 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4} I0720 00:44:09.009927 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000354 I0720 00:44:09.119652 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.109s user 0.018s sys 0.001s Metrics: {"bytes_written":8867,"cfile_init":1,"data dirs.queue_time_us":690,"data dirs.run_cpu_time_us":130,"data dirs.run_wall_time_us":1235,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":739,"lbm_writes_lt_1ms":23,"rows_written":161,"thread_start_us":347,"threads_started":1} I0720 00:44:09.121078 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=1.011587 I0720 00:44:09.151859 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.030s user 0.020s sys 0.004s Metrics: {"bytes_written":10724,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4466,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13200,"cfile_init":6,"data dirs.queue_time_us":328,"data dirs.run_cpu_time_us":2177,"data dirs.run_wall_time_us":2583,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":881,"lbm_reads_lt_1ms":50,"lbm_write_time_us":1239,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":233} I0720 00:44:09.153121 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 2360 bytes on disk I0720 00:44:09.154544 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0720 00:44:09.155413 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000381 I0720 00:44:09.198673 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.218s user 0.021s sys 0.000s Metrics: {"bytes_written":10613,"cfile_init":1,"data dirs.queue_time_us":70,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":608,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":727,"lbm_writes_lt_1ms":23,"rows_written":231} I0720 00:44:09.199718 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 2330 bytes on disk I0720 00:44:09.202595 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4} I0720 00:44:09.203343 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 1970 bytes on disk I0720 00:44:09.203912 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:09.204658 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 995 bytes on disk I0720 00:44:09.205476 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4} I0720 00:44:09.206221 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000722 I0720 00:44:09.276599 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.121s user 0.017s sys 0.000s Metrics: {"bytes_written":8707,"cfile_init":1,"data dirs.queue_time_us":765,"data dirs.run_cpu_time_us":116,"data dirs.run_wall_time_us":560,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"rows_written":156} I0720 00:44:09.277781 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 1631 bytes on disk I0720 00:44:09.282459 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":4146,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0720 00:44:09.283267 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.993469 I0720 00:44:09.346132 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.509s user 0.021s sys 0.035s Metrics: {"bytes_written":8367,"cfile_init":1,"data dirs.queue_time_us":136089,"data dirs.run_cpu_time_us":151,"data dirs.run_wall_time_us":669,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":879,"lbm_writes_lt_1ms":23,"rows_written":144,"thread_start_us":115885,"threads_started":1} I0720 00:44:09.347873 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=1.011611 W0720 00:44:09.376181 5435 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:39833, user_credentials={real_user=slave}} blocked reactor thread for 115011us W0720 00:44:09.402832 5684 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:39833, user_credentials={real_user=slave}} blocked reactor thread for 48547.1us I0720 00:44:09.437639 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.231s user 0.020s sys 0.000s Metrics: {"bytes_written":11153,"cfile_init":1,"data dirs.queue_time_us":1950,"data dirs.run_cpu_time_us":165,"data dirs.run_wall_time_us":1064,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":718,"lbm_writes_lt_1ms":23,"rows_written":251} I0720 00:44:09.443744 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.160s user 0.019s sys 0.002s Metrics: {"bytes_written":10562,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4354,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12850,"cfile_init":5,"data dirs.queue_time_us":370,"data dirs.run_cpu_time_us":496,"data dirs.run_wall_time_us":1024,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":912,"lbm_reads_lt_1ms":46,"lbm_write_time_us":771,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":227} I0720 00:44:09.445134 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 2318 bytes on disk I0720 00:44:09.446038 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:09.446923 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000446 I0720 00:44:09.449275 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 2519 bytes on disk I0720 00:44:09.450644 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:09.451395 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000787 I0720 00:44:09.493330 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.046s user 0.016s sys 0.000s Metrics: {"bytes_written":9286,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":138,"data dirs.run_wall_time_us":920,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"rows_written":177} I0720 00:44:09.494693 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 1812 bytes on disk I0720 00:44:09.498647 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0720 00:44:09.499406 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc): perf score=1.007865 I0720 00:44:09.637212 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.137s user 0.024s sys 0.000s Metrics: {"bytes_written":11591,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5106,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15086,"cfile_init":5,"data dirs.queue_time_us":5582,"data dirs.run_cpu_time_us":499,"data dirs.run_wall_time_us":1064,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":909,"lbm_reads_lt_1ms":46,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":268} I0720 00:44:09.639334 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 2671 bytes on disk I0720 00:44:09.639897 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:09.640667 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000463 I0720 00:44:09.659708 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":10202,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":380,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"rows_written":214} I0720 00:44:09.660837 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 2187 bytes on disk I0720 00:44:09.661309 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:09.661999 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86): perf score=1.012100 I0720 00:44:09.691756 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.030s user 0.023s sys 0.000s Metrics: {"bytes_written":13036,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6128,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17924,"cfile_init":5,"data dirs.queue_time_us":329,"data dirs.run_cpu_time_us":503,"data dirs.run_wall_time_us":900,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":876,"lbm_reads_lt_1ms":46,"lbm_write_time_us":601,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":324} I0720 00:44:09.692991 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 3186 bytes on disk I0720 00:44:09.693522 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:09.694402 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000458 I0720 00:44:09.735092 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.225s user 0.024s sys 0.000s Metrics: {"bytes_written":12597,"cfile_init":1,"data dirs.queue_time_us":2559,"data dirs.run_cpu_time_us":150,"data dirs.run_wall_time_us":1083,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5774,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":305,"thread_start_us":11619,"threads_started":1} I0720 00:44:09.735446 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.041s user 0.015s sys 0.003s Metrics: {"bytes_written":10381,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":93,"data dirs.run_wall_time_us":612,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"rows_written":221} I0720 00:44:09.736263 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 3009 bytes on disk I0720 00:44:09.736603 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 2215 bytes on disk I0720 00:44:09.736789 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:09.737087 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:09.737798 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=1.012542 I0720 00:44:09.750115 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000865 I0720 00:44:09.763218 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.025s user 0.022s sys 0.000s Metrics: {"bytes_written":13098,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6167,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18121,"cfile_init":5,"data dirs.queue_time_us":589,"data dirs.run_cpu_time_us":461,"data dirs.run_wall_time_us":2597,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":759,"lbm_reads_lt_1ms":46,"lbm_write_time_us":716,"lbm_writes_lt_1ms":23,"mutex_wait_us":41,"num_input_rowsets":2,"rows_written":326} I0720 00:44:09.764230 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 3198 bytes on disk I0720 00:44:09.764695 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:09.765377 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000458 I0720 00:44:09.784783 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.019s user 0.018s sys 0.000s Metrics: {"bytes_written":10199,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":916,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":23,"rows_written":213} I0720 00:44:09.786002 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 2163 bytes on disk I0720 00:44:09.786554 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0720 00:44:09.787309 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=1.008034 I0720 00:44:09.813736 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.026s user 0.021s sys 0.003s Metrics: {"bytes_written":12999,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6094,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17936,"cfile_init":5,"data dirs.queue_time_us":378,"data dirs.run_cpu_time_us":530,"data dirs.run_wall_time_us":931,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":916,"lbm_reads_lt_1ms":46,"lbm_write_time_us":739,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":322} I0720 00:44:09.814940 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 3209 bytes on disk I0720 00:44:09.815467 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:09.816195 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000385 I0720 00:44:09.841071 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.025s user 0.016s sys 0.000s Metrics: {"bytes_written":9169,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":416,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"rows_written":175} I0720 00:44:09.842306 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 1813 bytes on disk I0720 00:44:09.842845 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":12032} I0720 00:44:09.843582 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.994315 I0720 00:44:09.867297 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.024s user 0.018s sys 0.004s Metrics: {"bytes_written":12613,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5836,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17064,"cfile_init":5,"data dirs.queue_time_us":338,"data dirs.run_cpu_time_us":572,"data dirs.run_wall_time_us":924,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":901,"lbm_reads_lt_1ms":46,"lbm_write_time_us":762,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":308} I0720 00:44:09.868489 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000349 I0720 00:44:09.888481 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":9585,"cfile_init":1,"data dirs.queue_time_us":37,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":477,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"rows_written":191} I0720 00:44:09.889662 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743): perf score=1.012998 I0720 00:44:09.914510 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.025s user 0.017s sys 0.007s Metrics: {"bytes_written":13843,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6673,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19491,"cfile_init":6,"data dirs.queue_time_us":363,"data dirs.run_cpu_time_us":557,"data dirs.run_wall_time_us":1016,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1003,"lbm_reads_lt_1ms":50,"lbm_write_time_us":765,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":354} I0720 00:44:09.915673 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 3475 bytes on disk I0720 00:44:09.916170 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:09.916945 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000352 I0720 00:44:09.934376 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.017s user 0.013s sys 0.004s Metrics: {"bytes_written":9144,"cfile_init":1,"data dirs.queue_time_us":38,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":744,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":740,"lbm_writes_lt_1ms":23,"rows_written":173} I0720 00:44:09.935523 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 3081 bytes on disk I0720 00:44:09.935999 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:09.936710 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=1.009028 I0720 00:44:09.974381 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.037s user 0.024s sys 0.000s Metrics: {"bytes_written":13902,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6751,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19853,"cfile_init":6,"data dirs.queue_time_us":579,"data dirs.run_cpu_time_us":529,"data dirs.run_wall_time_us":938,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":969,"lbm_reads_lt_1ms":50,"lbm_write_time_us":760,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":358,"spinlock_wait_cycles":46848} I0720 00:44:09.975538 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 3523 bytes on disk I0720 00:44:09.992910 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:09.994789 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000291 I0720 00:44:10.013603 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.018s user 0.011s sys 0.004s Metrics: {"bytes_written":8246,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":582,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"rows_written":137} I0720 00:44:10.014147 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.664s user 0.025s sys 0.004s Metrics: {"bytes_written":10015,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4006,"cfile_cache_miss":26,"cfile_cache_miss_bytes":11764,"cfile_init":6,"data dirs.queue_time_us":221491,"data dirs.run_cpu_time_us":540,"data dirs.run_wall_time_us":1514,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":2470,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":1041,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":208,"thread_start_us":211583,"threads_started":3} I0720 00:44:10.014964 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=1.011739 I0720 00:44:10.015460 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 2120 bytes on disk I0720 00:44:10.016104 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:10.016970 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000617 I0720 00:44:10.083235 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.042s user 0.022s sys 0.004s Metrics: {"bytes_written":13680,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6639,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19439,"cfile_init":6,"data dirs.queue_time_us":342,"data dirs.run_cpu_time_us":575,"data dirs.run_wall_time_us":942,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1003,"lbm_reads_lt_1ms":50,"lbm_write_time_us":818,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":352} I0720 00:44:10.084830 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 3472 bytes on disk I0720 00:44:10.086735 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4} I0720 00:44:10.278774 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.262s user 0.022s sys 0.000s Metrics: {"bytes_written":11986,"cfile_init":1,"data dirs.queue_time_us":2045,"data dirs.run_cpu_time_us":164,"data dirs.run_wall_time_us":1052,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"rows_written":283} I0720 00:44:10.280239 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.969779 W0720 00:44:10.311879 5682 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:34347, user_credentials={real_user=slave}} blocked reactor thread for 46960.5us I0720 00:44:10.334308 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.583s user 0.021s sys 0.016s Metrics: {"bytes_written":13252,"cfile_init":1,"data dirs.queue_time_us":141249,"data dirs.run_cpu_time_us":212,"data dirs.run_wall_time_us":1274,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":720,"lbm_writes_lt_1ms":23,"rows_written":332,"thread_start_us":50503,"threads_started":1} I0720 00:44:10.335532 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 3286 bytes on disk I0720 00:44:10.336161 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4} I0720 00:44:10.336936 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.001029 I0720 00:44:10.338312 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000338 I0720 00:44:10.371803 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.033s user 0.017s sys 0.000s Metrics: {"bytes_written":9367,"cfile_init":1,"data dirs.queue_time_us":62,"data dirs.run_cpu_time_us":177,"data dirs.run_wall_time_us":645,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"rows_written":182} I0720 00:44:10.372910 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 1850 bytes on disk I0720 00:44:10.380178 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4} I0720 00:44:10.391417 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=1.011686 I0720 00:44:10.427381 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.032s user 0.023s sys 0.004s Metrics: {"bytes_written":15319,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7790,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22770,"cfile_init":5,"data dirs.queue_time_us":306,"data dirs.run_cpu_time_us":583,"data dirs.run_wall_time_us":1018,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1170,"lbm_reads_lt_1ms":46,"lbm_write_time_us":791,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":415} I0720 00:44:10.428573 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000320 I0720 00:44:10.450139 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.021s user 0.015s sys 0.003s Metrics: {"bytes_written":9109,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":526,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"rows_written":171} I0720 00:44:10.451241 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=1.011875 I0720 00:44:10.459017 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.178s user 0.019s sys 0.011s Metrics: {"bytes_written":13832,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6674,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19492,"cfile_init":6,"data dirs.queue_time_us":10683,"data dirs.run_cpu_time_us":506,"data dirs.run_wall_time_us":935,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1106,"lbm_reads_lt_1ms":50,"lbm_write_time_us":4828,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":354,"thread_start_us":7908,"threads_started":1} I0720 00:44:10.459966 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000736 I0720 00:44:10.480384 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.029s user 0.025s sys 0.001s Metrics: {"bytes_written":14910,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7476,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21990,"cfile_init":6,"data dirs.queue_time_us":437,"data dirs.run_cpu_time_us":529,"data dirs.run_wall_time_us":976,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":972,"lbm_reads_lt_1ms":50,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":398} I0720 00:44:10.482127 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 4036 bytes on disk I0720 00:44:10.483549 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.023s user 0.019s sys 0.001s Metrics: {"bytes_written":12858,"cfile_init":1,"data dirs.queue_time_us":2826,"data dirs.run_cpu_time_us":112,"data dirs.run_wall_time_us":1077,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"rows_written":318} I0720 00:44:10.484237 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4} I0720 00:44:10.484681 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.991361 I0720 00:44:10.484911 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 3904 bytes on disk I0720 00:44:10.486615 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:10.512876 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.028s user 0.021s sys 0.004s Metrics: {"bytes_written":15452,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7825,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22827,"cfile_init":6,"data dirs.queue_time_us":359,"data dirs.run_cpu_time_us":564,"data dirs.run_wall_time_us":1112,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":989,"lbm_reads_lt_1ms":50,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":417} I0720 00:44:10.514163 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000745 I0720 00:44:10.541409 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.027s user 0.019s sys 0.004s Metrics: {"bytes_written":13440,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":557,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"rows_written":339} I0720 00:44:10.542682 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 4087 bytes on disk I0720 00:44:10.544200 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:10.545181 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 3489 bytes on disk I0720 00:44:10.546347 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4} I0720 00:44:10.547053 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.991231 I0720 00:44:10.599536 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.050s user 0.026s sys 0.000s Metrics: {"bytes_written":15244,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7696,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22450,"cfile_init":6,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":478,"data dirs.run_wall_time_us":933,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1014,"lbm_reads_lt_1ms":50,"lbm_write_time_us":714,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":410} I0720 00:44:10.600742 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000765 I0720 00:44:10.638619 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.037s user 0.022s sys 0.002s Metrics: {"bytes_written":13806,"cfile_init":1,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":624,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":648,"lbm_writes_lt_1ms":23,"rows_written":353} I0720 00:44:10.639624 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 4007 bytes on disk I0720 00:44:10.640305 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:10.641172 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 3483 bytes on disk I0720 00:44:10.641645 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4} I0720 00:44:10.642191 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=1.006580 I0720 00:44:10.694757 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.052s user 0.017s sys 0.008s Metrics: {"bytes_written":15744,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8100,"cfile_cache_miss":26,"cfile_cache_miss_bytes":23600,"cfile_init":5,"data dirs.queue_time_us":331,"data dirs.run_cpu_time_us":512,"data dirs.run_wall_time_us":932,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":793,"lbm_reads_lt_1ms":46,"lbm_write_time_us":675,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":432} I0720 00:44:10.695942 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000786 I0720 00:44:10.724663 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.028s user 0.019s sys 0.005s Metrics: {"bytes_written":13467,"cfile_init":1,"data dirs.queue_time_us":75,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":651,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":23,"rows_written":340} I0720 00:44:10.725731 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 4198 bytes on disk I0720 00:44:10.726475 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:10.727367 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 3367 bytes on disk I0720 00:44:10.730314 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:10.731122 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=1.011182 I0720 00:44:10.739006 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000346 I0720 00:44:10.761564 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.029s user 0.025s sys 0.000s Metrics: {"bytes_written":16145,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8336,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24428,"cfile_init":5,"data dirs.queue_time_us":314,"data dirs.run_cpu_time_us":500,"data dirs.run_wall_time_us":889,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":888,"lbm_reads_lt_1ms":46,"lbm_write_time_us":823,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":445} I0720 00:44:10.763478 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 4334 bytes on disk I0720 00:44:10.765393 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:10.766438 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000799 I0720 00:44:10.801264 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.032s user 0.024s sys 0.000s Metrics: {"bytes_written":14096,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":528,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":794,"lbm_writes_lt_1ms":23,"rows_written":365} I0720 00:44:10.802755 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 3597 bytes on disk I0720 00:44:10.803398 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:10.805357 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743): perf score=1.012731 I0720 00:44:10.840445 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.032s user 0.028s sys 0.000s Metrics: {"bytes_written":16574,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8667,"cfile_cache_miss":26,"cfile_cache_miss_bytes":25271,"cfile_init":5,"data dirs.queue_time_us":314,"data dirs.run_cpu_time_us":563,"data dirs.run_wall_time_us":1067,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1412,"lbm_reads_lt_1ms":46,"lbm_write_time_us":777,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":463} I0720 00:44:10.842468 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000786 I0720 00:44:10.845705 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.509s user 0.025s sys 0.006s Metrics: {"bytes_written":15124,"cfile_init":1,"data dirs.queue_time_us":18065,"data dirs.run_cpu_time_us":167,"data dirs.run_wall_time_us":819,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19101,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":407,"thread_start_us":29932,"threads_started":1} I0720 00:44:10.847021 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.001170 I0720 00:44:10.864091 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.125s user 0.014s sys 0.004s Metrics: {"bytes_written":9743,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":112,"data dirs.run_wall_time_us":564,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1038,"lbm_writes_lt_1ms":23,"rows_written":196} I0720 00:44:10.865576 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc): perf score=1.013412 I0720 00:44:10.956645 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.114s user 0.022s sys 0.003s Metrics: {"bytes_written":14490,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":498,"drs_written":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4,"lbm_write_time_us":773,"lbm_writes_lt_1ms":23,"rows_written":381} I0720 00:44:10.964280 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.098s user 0.027s sys 0.000s Metrics: {"bytes_written":16564,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8679,"cfile_cache_miss":26,"cfile_cache_miss_bytes":25309,"cfile_init":6,"data dirs.queue_time_us":445,"data dirs.run_cpu_time_us":603,"data dirs.run_wall_time_us":1404,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1132,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2383,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":464} I0720 00:44:10.965605 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000363 I0720 00:44:10.966936 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86): perf score=1.011225 I0720 00:44:10.987522 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":9515,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":616,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"rows_written":187} I0720 00:44:10.988875 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 4481 bytes on disk I0720 00:44:10.990901 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4} I0720 00:44:10.991739 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86): perf score=1.014274 I0720 00:44:11.031291 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.039s user 0.026s sys 0.002s Metrics: {"bytes_written":17790,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9538,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27730,"cfile_init":6,"data dirs.queue_time_us":307,"data dirs.run_cpu_time_us":545,"data dirs.run_wall_time_us":1202,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":3120,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":786,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":511} I0720 00:44:11.032836 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000360 I0720 00:44:11.053982 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.021s user 0.007s sys 0.007s Metrics: {"bytes_written":9188,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":558,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"rows_written":175} I0720 00:44:11.055392 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=1.005498 I0720 00:44:11.103030 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.256s user 0.031s sys 0.005s Metrics: {"bytes_written":17380,"cfile_init":1,"data dirs.queue_time_us":5101,"data dirs.run_cpu_time_us":138,"data dirs.run_wall_time_us":34171,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"rows_written":495,"thread_start_us":7380,"threads_started":1} I0720 00:44:11.109920 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 4774 bytes on disk I0720 00:44:11.110553 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:11.111510 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 3950 bytes on disk I0720 00:44:11.114486 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:11.115288 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000873 I0720 00:44:11.142531 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.087s user 0.021s sys 0.008s Metrics: {"bytes_written":17546,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9358,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27126,"cfile_init":6,"data dirs.queue_time_us":334,"data dirs.run_cpu_time_us":499,"data dirs.run_wall_time_us":965,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":8107,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":1062,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":501} I0720 00:44:11.144474 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000378 I0720 00:44:11.166604 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.022s user 0.019s sys 0.000s Metrics: {"bytes_written":9185,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":118,"data dirs.run_wall_time_us":554,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"rows_written":176} I0720 00:44:11.168035 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=1.006484 I0720 00:44:11.204232 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.035s user 0.025s sys 0.003s Metrics: {"bytes_written":17491,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9302,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27076,"cfile_init":6,"data dirs.queue_time_us":419,"data dirs.run_cpu_time_us":511,"data dirs.run_wall_time_us":972,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1006,"lbm_reads_lt_1ms":50,"lbm_write_time_us":772,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":498} I0720 00:44:11.205729 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 4951 bytes on disk I0720 00:44:11.207746 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:11.208747 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.241s user 0.023s sys 0.007s Metrics: {"bytes_written":17380,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9251,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26861,"cfile_init":6,"data dirs.queue_time_us":20578,"data dirs.run_cpu_time_us":544,"data dirs.run_wall_time_us":1149,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1024,"lbm_reads_lt_1ms":50,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":495} I0720 00:44:11.209455 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000380 I0720 00:44:11.209705 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 4774 bytes on disk I0720 00:44:11.210275 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:11.211073 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 4472 bytes on disk I0720 00:44:11.211792 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:11.212538 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000880 W0720 00:44:11.218268 5682 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:34347, user_credentials={real_user=slave}} blocked reactor thread for 62049us I0720 00:44:11.244019 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.129s user 0.028s sys 0.002s Metrics: {"bytes_written":14851,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":539,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"rows_written":396} I0720 00:44:11.246913 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 3912 bytes on disk I0720 00:44:11.247624 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:11.250707 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.989481 I0720 00:44:11.271778 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.062s user 0.013s sys 0.004s Metrics: {"bytes_written":9479,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":719,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"rows_written":185} I0720 00:44:11.273710 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=1.010819 I0720 00:44:11.287158 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.074s user 0.025s sys 0.003s Metrics: {"bytes_written":15271,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":554,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"rows_written":410} I0720 00:44:11.290027 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 3960 bytes on disk I0720 00:44:11.290700 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4} I0720 00:44:11.292668 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=1.012773 I0720 00:44:11.299058 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.132:0 --local_ip_for_outbound_sockets=127.0.58.132 --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:38255 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:11.318986 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.045s user 0.028s sys 0.000s Metrics: {"bytes_written":17329,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9212,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26924,"cfile_init":6,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":1664,"data dirs.run_wall_time_us":2087,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":13164,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":1895,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":493} I0720 00:44:11.320394 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000398 I0720 00:44:11.330869 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.080s user 0.036s sys 0.003s Metrics: {"bytes_written":17089,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9010,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26090,"cfile_init":5,"data dirs.queue_time_us":7331,"data dirs.run_cpu_time_us":639,"data dirs.run_wall_time_us":4713,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":900,"lbm_reads_lt_1ms":46,"lbm_write_time_us":653,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":482,"thread_start_us":6562,"threads_started":2} I0720 00:44:11.332185 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000848 I0720 00:44:11.336066 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.015s user 0.009s sys 0.004s Metrics: {"bytes_written":8773,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":533,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":23,"rows_written":160} I0720 00:44:11.337440 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 4826 bytes on disk I0720 00:44:11.338274 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":233,"lbm_reads_lt_1ms":4} I0720 00:44:11.339102 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743): perf score=1.008535 I0720 00:44:11.359019 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.066s user 0.029s sys 0.001s Metrics: {"bytes_written":17909,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9577,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27725,"cfile_init":5,"data dirs.queue_time_us":5748,"data dirs.run_cpu_time_us":578,"data dirs.run_wall_time_us":1023,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":736,"lbm_reads_lt_1ms":46,"lbm_write_time_us":915,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":513} I0720 00:44:11.360371 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000915 I0720 00:44:11.367444 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.028s user 0.027s sys 0.000s Metrics: {"bytes_written":17918,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9594,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27882,"cfile_init":6,"data dirs.queue_time_us":474,"data dirs.run_cpu_time_us":475,"data dirs.run_wall_time_us":942,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":940,"lbm_reads_lt_1ms":50,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":514,"spinlock_wait_cycles":3456} I0720 00:44:11.368665 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 4979 bytes on disk I0720 00:44:11.369441 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4} I0720 00:44:11.370345 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000399 I0720 00:44:11.386785 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.016s user 0.010s sys 0.006s Metrics: {"bytes_written":8877,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":111,"data dirs.run_wall_time_us":490,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"rows_written":161} I0720 00:44:11.388021 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.989510 I0720 00:44:11.402717 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.070s user 0.024s sys 0.008s Metrics: {"bytes_written":15330,"cfile_init":1,"data dirs.queue_time_us":4092,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":596,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"rows_written":414} I0720 00:44:11.404192 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 4682 bytes on disk I0720 00:44:11.404938 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:11.406136 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=1.013864 I0720 00:44:11.417121 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.056s user 0.021s sys 0.005s Metrics: {"bytes_written":15523,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":96,"data dirs.run_wall_time_us":618,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3346,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":420} I0720 00:44:11.417204 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.029s user 0.023s sys 0.005s Metrics: {"bytes_written":18023,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9689,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28195,"cfile_init":6,"data dirs.queue_time_us":407,"data dirs.run_cpu_time_us":350,"data dirs.run_wall_time_us":840,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":835,"lbm_reads_lt_1ms":50,"lbm_write_time_us":826,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":519} I0720 00:44:11.418260 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 5037 bytes on disk I0720 00:44:11.418511 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc): perf score=1.010228 I0720 00:44:11.418779 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:11.419365 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000390 I0720 00:44:11.432200 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.013s user 0.000s sys 0.010s Metrics: {"bytes_written":8887,"cfile_init":1,"data dirs.queue_time_us":98,"data dirs.run_cpu_time_us":147,"data dirs.run_wall_time_us":676,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":495,"lbm_writes_lt_1ms":23,"rows_written":163} I0720 00:44:11.433092 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 4809 bytes on disk I0720 00:44:11.433562 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4} I0720 00:44:11.434120 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=1.012767 I0720 00:44:11.457311 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.023s user 0.014s sys 0.008s Metrics: {"bytes_written":17885,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9610,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27770,"cfile_init":6,"data dirs.queue_time_us":392,"data dirs.run_cpu_time_us":486,"data dirs.run_wall_time_us":909,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":672,"lbm_reads_lt_1ms":50,"lbm_write_time_us":646,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":515} I0720 00:44:11.458575 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 5006 bytes on disk I0720 00:44:11.461313 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:11.462317 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000303 I0720 00:44:11.463232 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.057s user 0.030s sys 0.000s Metrics: {"bytes_written":17694,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9431,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27371,"cfile_init":6,"data dirs.queue_time_us":338,"data dirs.run_cpu_time_us":534,"data dirs.run_wall_time_us":951,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":9037,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":785,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":505} I0720 00:44:11.464442 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 4917 bytes on disk I0720 00:44:11.465024 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:11.465790 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000776 I0720 00:44:11.478440 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":7458,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":549,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":730,"lbm_writes_lt_1ms":23,"rows_written":107} I0720 00:44:11.479756 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=1.014378 I0720 00:44:11.483338 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.065s user 0.030s sys 0.001s Metrics: {"bytes_written":18596,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10105,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29329,"cfile_init":6,"data dirs.queue_time_us":327,"data dirs.run_cpu_time_us":460,"data dirs.run_wall_time_us":922,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":937,"lbm_reads_lt_1ms":50,"lbm_write_time_us":735,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":542} I0720 00:44:11.484854 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 5241 bytes on disk I0720 00:44:11.485476 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:11.486274 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 4929 bytes on disk I0720 00:44:11.486747 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:11.487418 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=0.000595 I0720 00:44:11.496402 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.030s user 0.019s sys 0.010s Metrics: {"bytes_written":13924,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":85,"data dirs.run_wall_time_us":506,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":662,"lbm_writes_lt_1ms":23,"rows_written":359} I0720 00:44:11.497444 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 3513 bytes on disk I0720 00:44:11.498169 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:11.498802 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=1.013170 I0720 00:44:11.517005 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.037s user 0.025s sys 0.004s Metrics: {"bytes_written":18109,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9740,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28140,"cfile_init":6,"data dirs.queue_time_us":559,"data dirs.run_cpu_time_us":537,"data dirs.run_wall_time_us":1001,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":923,"lbm_reads_lt_1ms":50,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":522,"spinlock_wait_cycles":3840} I0720 00:44:11.518193 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000297 I0720 00:44:11.529887 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.031s user 0.026s sys 0.003s Metrics: {"bytes_written":18109,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9740,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28216,"cfile_init":5,"data dirs.queue_time_us":701,"data dirs.run_cpu_time_us":516,"data dirs.run_wall_time_us":968,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":940,"lbm_reads_lt_1ms":46,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":522} I0720 00:44:11.530871 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 5045 bytes on disk I0720 00:44:11.531446 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:11.532061 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000709 I0720 00:44:11.532335 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.045s user 0.018s sys 0.000s Metrics: {"bytes_written":12815,"cfile_init":1,"data dirs.queue_time_us":107,"data dirs.run_cpu_time_us":131,"data dirs.run_wall_time_us":531,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":513,"lbm_writes_lt_1ms":23,"rows_written":314} I0720 00:44:11.533234 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 3086 bytes on disk I0720 00:44:11.533247 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.015s user 0.012s sys 0.000s Metrics: {"bytes_written":7455,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":130,"data dirs.run_wall_time_us":590,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":732,"lbm_writes_lt_1ms":23,"rows_written":107} I0720 00:44:11.533749 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:11.534337 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489): perf score=1.013379 I0720 00:44:11.534425 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.987270 I0720 00:44:11.557332 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.025s user 0.021s sys 0.003s Metrics: {"bytes_written":12990,"cfile_init":1,"data dirs.queue_time_us":59,"data dirs.run_cpu_time_us":168,"data dirs.run_wall_time_us":582,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"rows_written":321} I0720 00:44:11.558441 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=1.008251 I0720 00:44:11.563246 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.029s user 0.015s sys 0.012s Metrics: {"bytes_written":17694,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9431,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27353,"cfile_init":6,"data dirs.queue_time_us":697,"data dirs.run_cpu_time_us":544,"data dirs.run_wall_time_us":993,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":962,"lbm_reads_lt_1ms":50,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":505} I0720 00:44:11.564469 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 5045 bytes on disk I0720 00:44:11.564992 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:11.565652 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 4917 bytes on disk I0720 00:44:11.566140 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:11.566764 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 4782 bytes on disk I0720 00:44:11.567198 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:11.571939 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.037s user 0.018s sys 0.003s Metrics: {"bytes_written":18109,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9740,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28290,"cfile_init":5,"data dirs.queue_time_us":1061,"data dirs.run_cpu_time_us":500,"data dirs.run_wall_time_us":884,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":574,"lbm_reads_lt_1ms":46,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"mutex_wait_us":18,"num_input_rowsets":2,"rows_written":522,"spinlock_wait_cycles":16768} I0720 00:44:11.572917 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489): 5045 bytes on disk I0720 00:44:11.573480 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(d54fc0a6131248cc929ba6d4b0a17489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:11.574108 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000360 I0720 00:44:11.588610 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.030s user 0.020s sys 0.008s Metrics: {"bytes_written":17779,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9560,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27888,"cfile_init":6,"data dirs.queue_time_us":374,"data dirs.run_cpu_time_us":529,"data dirs.run_wall_time_us":916,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":988,"lbm_reads_lt_1ms":50,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":512} I0720 00:44:11.589582 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000664 I0720 00:44:11.591295 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.017s user 0.011s sys 0.004s Metrics: {"bytes_written":8864,"cfile_init":1,"data dirs.queue_time_us":109,"data dirs.run_cpu_time_us":165,"data dirs.run_wall_time_us":589,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"rows_written":161} I0720 00:44:11.592283 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 1654 bytes on disk I0720 00:44:11.592909 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:11.593490 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=1.012798 I0720 00:44:11.613842 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.024s user 0.021s sys 0.000s Metrics: {"bytes_written":11989,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":139,"data dirs.run_wall_time_us":663,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"rows_written":283} I0720 00:44:11.614841 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743): perf score=1.012980 I0720 00:44:11.624121 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.030s user 0.025s sys 0.003s Metrics: {"bytes_written":17885,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9616,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28020,"cfile_init":5,"data dirs.queue_time_us":555,"data dirs.run_cpu_time_us":627,"data dirs.run_wall_time_us":1107,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":894,"lbm_reads_lt_1ms":46,"lbm_write_time_us":1105,"lbm_writes_lt_1ms":23,"mutex_wait_us":22,"num_input_rowsets":2,"rows_written":515} I0720 00:44:11.625093 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000317 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:11.634119 6005 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:11.634398 6005 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:11.636984 6005 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:11.637095 6005 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:11.637135 6005 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:11.637210 6005 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.132 I0720 00:44:11.637526 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.012s user 0.009s sys 0.000s Metrics: {"bytes_written":6668,"cfile_init":1,"data dirs.queue_time_us":147,"data dirs.run_cpu_time_us":116,"data dirs.run_wall_time_us":559,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"rows_written":78} I0720 00:44:11.638535 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 5006 bytes on disk I0720 00:44:11.639124 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:11.639698 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 884 bytes on disk I0720 00:44:11.640216 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:11.640718 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.983911 I0720 00:44:11.645884 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.031s user 0.026s sys 0.002s Metrics: {"bytes_written":17918,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9594,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27942,"cfile_init":6,"data dirs.queue_time_us":518,"data dirs.run_cpu_time_us":547,"data dirs.run_wall_time_us":982,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":889,"lbm_reads_lt_1ms":50,"lbm_write_time_us":715,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":514} I0720 00:44:11.646772 6005 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --webserver_interface=127.0.58.132 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:38255 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.132 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:11.646816 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=0.000614 I0720 00:44:11.654644 6005 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 87016us I0720 00:44:11.655329 6005 fs_manager.cc:263] Metadata directory not provided I0720 00:44:11.655411 6005 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:11.655699 6005 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:11.655777 6005 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:11.658133 6005 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/instance: uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" format_stamp: "Formatted at 2019-07-20 00:44:11 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:11.658763 6005 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-1/instance: uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" format_stamp: "Formatted at 2019-07-20 00:44:11 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:11.659204 6005 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-2/instance: uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" format_stamp: "Formatted at 2019-07-20 00:44:11 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:11.659601 6005 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/wal/instance: uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" format_stamp: "Formatted at 2019-07-20 00:44:11 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:11.668212 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.021s user 0.019s sys 0.000s Metrics: {"bytes_written":11446,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":96,"data dirs.run_wall_time_us":558,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":609,"lbm_writes_lt_1ms":23,"rows_written":264} I0720 00:44:11.669198 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 4979 bytes on disk I0720 00:44:11.669271 6005 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.004s I0720 00:44:11.669788 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:11.670424 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e): perf score=1.009150 I0720 00:44:11.670755 6005 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:11.671314 6005 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:11.672236 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.031s user 0.023s sys 0.005s Metrics: {"bytes_written":17346,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9251,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26899,"cfile_init":5,"data dirs.queue_time_us":886,"data dirs.run_cpu_time_us":553,"data dirs.run_wall_time_us":1043,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":865,"lbm_reads_lt_1ms":46,"lbm_write_time_us":743,"lbm_writes_lt_1ms":23,"mutex_wait_us":113,"num_input_rowsets":2,"rows_written":495,"spinlock_wait_cycles":1408} I0720 00:44:11.673233 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.000314 I0720 00:44:11.676092 6005 fs_manager.cc:419] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s I0720 00:44:11.676169 6005 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/wal uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" format_stamp: "Formatted at 2019-07-20 00:44:11 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:11.676831 6005 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:11.687491 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":7113,"cfile_init":1,"data dirs.queue_time_us":80,"data dirs.run_cpu_time_us":112,"data dirs.run_wall_time_us":658,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"rows_written":95} I0720 00:44:11.688897 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): perf score=0.986390 I0720 00:44:11.699532 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.029s user 0.027s sys 0.000s Metrics: {"bytes_written":17885,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9610,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27888,"cfile_init":6,"data dirs.queue_time_us":1449,"data dirs.run_cpu_time_us":470,"data dirs.run_wall_time_us":910,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":850,"lbm_reads_lt_1ms":50,"lbm_write_time_us":1273,"lbm_writes_lt_1ms":23,"mutex_wait_us":28,"num_input_rowsets":2,"rows_written":515} I0720 00:44:11.700472 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e): 5006 bytes on disk I0720 00:44:11.701007 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(f0f53e18f4ec4eec823ef551f843b54e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:11.701694 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000562 I0720 00:44:11.711947 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.023s user 0.018s sys 0.003s Metrics: {"bytes_written":17694,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9431,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27307,"cfile_init":6,"data dirs.queue_time_us":760,"data dirs.run_cpu_time_us":429,"data dirs.run_wall_time_us":818,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":796,"lbm_reads_lt_1ms":50,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":505,"spinlock_wait_cycles":3840} I0720 00:44:11.712987 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000298 I0720 00:44:11.720602 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.019s user 0.017s sys 0.000s Metrics: {"bytes_written":10834,"cfile_init":1,"data dirs.queue_time_us":71,"data dirs.run_cpu_time_us":156,"data dirs.run_wall_time_us":586,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"rows_written":240} I0720 00:44:11.721715 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=1.006712 I0720 00:44:11.723450 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.010s user 0.008s sys 0.000s Metrics: {"bytes_written":6704,"cfile_init":1,"data dirs.queue_time_us":80,"data dirs.run_cpu_time_us":135,"data dirs.run_wall_time_us":581,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"rows_written":80} I0720 00:44:11.724488 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a): 4917 bytes on disk I0720 00:44:11.725078 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(26b0b6bdff8a4bc7be9e3eeff65c3b9a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:11.725901 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 4809 bytes on disk I0720 00:44:11.726434 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:11.727030 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.992393 I0720 00:44:11.742576 6005 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:11.747109 6005 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:11.747206 6005 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:11.751720 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.030s user 0.027s sys 0.002s Metrics: {"bytes_written":18719,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10161,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29497,"cfile_init":6,"data dirs.queue_time_us":594,"data dirs.run_cpu_time_us":558,"data dirs.run_wall_time_us":1054,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":929,"lbm_reads_lt_1ms":50,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":545,"spinlock_wait_cycles":20864} I0720 00:44:11.752769 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 5279 bytes on disk I0720 00:44:11.753334 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:11.753865 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000394 I0720 00:44:11.756922 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.030s user 0.022s sys 0.003s Metrics: {"bytes_written":17779,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9555,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27447,"cfile_init":6,"data dirs.queue_time_us":510,"data dirs.run_cpu_time_us":331,"data dirs.run_wall_time_us":744,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":892,"lbm_reads_lt_1ms":50,"lbm_write_time_us":655,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":512,"spinlock_wait_cycles":5888} I0720 00:44:11.758401 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.000287 I0720 00:44:11.772692 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.014s user 0.006s sys 0.004s Metrics: {"bytes_written":6627,"cfile_init":1,"data dirs.queue_time_us":93,"data dirs.run_cpu_time_us":121,"data dirs.run_wall_time_us":568,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"rows_written":74} I0720 00:44:11.773947 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.947735 I0720 00:44:11.774132 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.020s user 0.018s sys 0.000s Metrics: {"bytes_written":9532,"cfile_init":1,"data dirs.queue_time_us":80,"data dirs.run_cpu_time_us":123,"data dirs.run_wall_time_us":571,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"rows_written":187} I0720 00:44:11.775373 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905): perf score=0.991222 I0720 00:44:11.800734 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.026s user 0.024s sys 0.000s Metrics: {"bytes_written":18023,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9689,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28113,"cfile_init":6,"data dirs.queue_time_us":1668,"data dirs.run_cpu_time_us":428,"data dirs.run_wall_time_us":902,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1421,"lbm_reads_lt_1ms":50,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":519,"spinlock_wait_cycles":4992} I0720 00:44:11.801877 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.000278 I0720 00:44:11.811259 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.035s user 0.029s sys 0.003s Metrics: {"bytes_written":18023,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9684,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28218,"cfile_init":6,"data dirs.queue_time_us":470,"data dirs.run_cpu_time_us":361,"data dirs.run_wall_time_us":806,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":947,"lbm_reads_lt_1ms":50,"lbm_write_time_us":1398,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":519} I0720 00:44:11.811461 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: FlushMRSOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.009s user 0.007s sys 0.001s Metrics: {"bytes_written":5997,"cfile_init":1,"data dirs.queue_time_us":73,"data dirs.run_cpu_time_us":96,"data dirs.run_wall_time_us":585,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":405,"lbm_writes_lt_1ms":23,"rows_written":51,"spinlock_wait_cycles":14080} I0720 00:44:11.812395 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 5037 bytes on disk I0720 00:44:11.812461 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743): perf score=0.993978 I0720 00:44:11.813295 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:11.814013 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 4965 bytes on disk I0720 00:44:11.814493 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:11.827248 6005 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.132:44535 I0720 00:44:11.828377 6005 webserver.cc:374] Webserver started at http://127.0.58.132:46515/ using document root and password file I0720 00:44:11.828990 6005 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb I0720 00:44:11.831923 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 6005 I0720 00:44:11.841147 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: CompactRowSetsOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.028s user 0.021s sys 0.006s Metrics: {"bytes_written":17918,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9594,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27704,"cfile_init":6,"data dirs.queue_time_us":568,"data dirs.run_cpu_time_us":418,"data dirs.run_wall_time_us":806,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":965,"lbm_reads_lt_1ms":50,"lbm_write_time_us":577,"lbm_writes_lt_1ms":23,"mutex_wait_us":1,"num_input_rowsets":2,"rows_written":514,"spinlock_wait_cycles":1792} I0720 00:44:11.842308 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905): 5037 bytes on disk I0720 00:44:11.842890 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(736b3f88dd4a41fbb23ef072e4e28905) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:11.843505 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743): 4979 bytes on disk I0720 00:44:11.844251 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(c1e0c42e42524c22af51c8e4a457d743) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:11.844878 5545 maintenance_manager.cc:327] P be9af1eb9bb14a7e8ef0cf71154d5e6e: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 4965 bytes on disk I0720 00:44:11.845310 5439 maintenance_manager.cc:532] P be9af1eb9bb14a7e8ef0cf71154d5e6e: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} W0720 00:44:11.845527 5541 generic_service.cc:128] {username='slave'} at 127.0.0.1:52310 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:11.845644 5541 generic_service.cc:148] {username='slave'} at 127.0.0.1:52310 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:11.853854 6128 heartbeater.cc:324] Connected to a master server at 127.0.58.190:38255 I0720 00:44:11.854091 6128 heartbeater.cc:405] Registering TS with master... I0720 00:44:11.854645 6128 heartbeater.cc:454] Master 127.0.58.190:38255 requested a full tablet report, sending... I0720 00:44:11.855924 5374 ts_manager.cc:161] Registered new tserver with Master: b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535) I0720 00:44:11.857024 5374 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.132:57683 W0720 00:44:11.859288 5666 generic_service.cc:128] {username='slave'} at 127.0.0.1:42484 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:11.859396 5666 generic_service.cc:148] {username='slave'} at 127.0.0.1:42484 changed flags via RPC: enable_tablet_copy from 'true' to 'false' W0720 00:44:11.865778 5790 generic_service.cc:128] {username='slave'} at 127.0.0.1:49294 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:11.865878 5790 generic_service.cc:148] {username='slave'} at 127.0.0.1:49294 changed flags via RPC: enable_tablet_copy from 'true' to 'false' W0720 00:44:11.873155 6125 generic_service.cc:128] {username='slave'} at 127.0.0.1:57806 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:11.873255 6125 generic_service.cc:148] {username='slave'} at 127.0.0.1:57806 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:11.874634 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 5412 I0720 00:44:11.876247 5541 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:12.019181 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 5412 I0720 00:44:12.041859 234 disk_failure-itest.cc:273] Restarting server with injected errors... I0720 00:44:12.042336 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.129:34347 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_port=42415 --webserver_interface=127.0.58.129 --tserver_master_addrs=127.0.58.190:38255 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** --env_inject_eio=1.0 with env {} I0720 00:44:12.065888 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000339 I0720 00:44:12.068979 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(f50d69e061a145869a005de2b35d1acc): perf score=0.000335 I0720 00:44:12.087947 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.022s user 0.005s sys 0.008s Metrics: {"bytes_written":8148,"cfile_init":1,"data dirs.queue_time_us":113,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":527,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"rows_written":135} I0720 00:44:12.089112 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 1416 bytes on disk I0720 00:44:12.089632 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0720 00:44:12.090306 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc): perf score=1.013561 I0720 00:44:12.092571 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.023s user 0.007s sys 0.003s Metrics: {"bytes_written":6686,"cfile_init":1,"data dirs.queue_time_us":334,"data dirs.run_cpu_time_us":146,"data dirs.run_wall_time_us":606,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":607,"lbm_writes_lt_1ms":23,"rows_written":78,"thread_start_us":5530,"threads_started":1} I0720 00:44:12.093781 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc): perf score=1.007453 I0720 00:44:12.110713 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.020s user 0.020s sys 0.000s Metrics: {"bytes_written":18596,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10105,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29129,"cfile_init":5,"data dirs.queue_time_us":691,"data dirs.run_cpu_time_us":538,"data dirs.run_wall_time_us":926,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":701,"lbm_reads_lt_1ms":46,"lbm_write_time_us":475,"lbm_writes_lt_1ms":23,"mutex_wait_us":15,"num_input_rowsets":2,"rows_written":542,"spinlock_wait_cycles":3584} I0720 00:44:12.111552 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000280 I0720 00:44:12.120975 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.009s user 0.007s sys 0.000s Metrics: {"bytes_written":5595,"cfile_init":1,"data dirs.queue_time_us":95,"data dirs.run_cpu_time_us":153,"data dirs.run_wall_time_us":575,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":428,"lbm_writes_lt_1ms":23,"rows_written":36} I0720 00:44:12.121976 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86): perf score=1.003127 I0720 00:44:12.141161 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.019s user 0.019s sys 0.000s Metrics: {"bytes_written":18327,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9903,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28491,"cfile_init":6,"data dirs.queue_time_us":690,"data dirs.run_cpu_time_us":521,"data dirs.run_wall_time_us":1000,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":615,"lbm_reads_lt_1ms":50,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":531,"spinlock_wait_cycles":3712} I0720 00:44:12.141996 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 5241 bytes on disk I0720 00:44:12.142457 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} I0720 00:44:12.143002 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 5138 bytes on disk I0720 00:44:12.143376 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4} I0720 00:44:12.158696 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.056s user 0.026s sys 0.005s Metrics: {"bytes_written":18596,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10105,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29053,"cfile_init":6,"data dirs.queue_time_us":8395,"data dirs.run_cpu_time_us":583,"data dirs.run_wall_time_us":1006,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":883,"lbm_reads_lt_1ms":50,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":542,"spinlock_wait_cycles":6400,"thread_start_us":11342,"threads_started":2} I0720 00:44:12.159587 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc): 5241 bytes on disk I0720 00:44:12.160045 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(f50d69e061a145869a005de2b35d1acc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0720 00:44:12.160604 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.000326 I0720 00:44:12.168884 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.008s user 0.006s sys 0.000s Metrics: {"bytes_written":5156,"cfile_init":1,"data dirs.queue_time_us":90,"data dirs.run_cpu_time_us":91,"data dirs.run_wall_time_us":526,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":480,"lbm_writes_lt_1ms":23,"rows_written":20} I0720 00:44:12.169697 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86): perf score=0.990703 I0720 00:44:12.190755 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.021s user 0.017s sys 0.004s Metrics: {"bytes_written":18327,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9901,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28497,"cfile_init":6,"data dirs.queue_time_us":815,"data dirs.run_cpu_time_us":556,"data dirs.run_wall_time_us":979,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":692,"lbm_reads_lt_1ms":50,"lbm_write_time_us":542,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":531,"spinlock_wait_cycles":3328} I0720 00:44:12.191860 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.000316 I0720 00:44:12.199815 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.008s user 0.002s sys 0.004s Metrics: {"bytes_written":5810,"cfile_init":1,"data dirs.queue_time_us":83,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":607,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"rows_written":44} I0720 00:44:12.200879 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86): 5138 bytes on disk I0720 00:44:12.201426 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(6c0c6e70945746faa55c4964f07a5a86) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:12.202148 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 535 bytes on disk I0720 00:44:12.202579 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} I0720 00:44:12.203202 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3): perf score=0.958637 W0720 00:44:12.215579 5558 consensus_peers.cc:458] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:44:12.218926 5558 consensus_peers.cc:458] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.224174 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.021s user 0.020s sys 0.000s Metrics: {"bytes_written":18719,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10161,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29209,"cfile_init":5,"data dirs.queue_time_us":860,"data dirs.run_cpu_time_us":558,"data dirs.run_wall_time_us":1003,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":619,"lbm_reads_lt_1ms":46,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"mutex_wait_us":17,"num_input_rowsets":2,"rows_written":545,"spinlock_wait_cycles":18048} I0720 00:44:12.225208 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3): 5279 bytes on disk I0720 00:44:12.225697 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(dff7af31a11e4cf78f6efbbfa12762d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} I0720 00:44:12.226364 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.000294 I0720 00:44:12.233896 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":4977,"cfile_init":1,"data dirs.queue_time_us":93,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":575,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":468,"lbm_writes_lt_1ms":23,"rows_written":14} I0720 00:44:12.234901 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 260 bytes on disk I0720 00:44:12.235349 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:12.235554 6138 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:12.235882 6138 flags.cc:404] Enabled unsafe flag: --never_fsync=true I0720 00:44:12.235970 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f): perf score=0.929943 W0720 00:44:12.238040 6138 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:12.238169 6138 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:12.238240 6138 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:12.238318 6138 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:44:12.251581 6138 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:34347 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=42415 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:38255 --env_inject_eio=1 --env_inject_eio_globs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:12.254633 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.018s user 0.014s sys 0.004s Metrics: {"bytes_written":17779,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9560,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27524,"cfile_init":5,"data dirs.queue_time_us":777,"data dirs.run_cpu_time_us":507,"data dirs.run_wall_time_us":1040,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":597,"lbm_reads_lt_1ms":46,"lbm_write_time_us":477,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":512,"spinlock_wait_cycles":28544} I0720 00:44:12.255652 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f): 4965 bytes on disk I0720 00:44:12.256147 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(43e3ab995f34471c92e887c28c2e2e0f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:12.259634 6138 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 87516us W0720 00:44:12.259928 6138 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:44:12.259976 6138 fault_injection.cc:45] THIS SERVER MAY CRASH! I0720 00:44:12.260488 6138 fs_manager.cc:263] Metadata directory not provided I0720 00:44:12.260581 6138 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory E0720 00:44:12.261876 6138 env_posix.cc:316] I/O error, context: INJECTED FAILURE E0720 00:44:12.263451 6138 env_posix.cc:316] I/O error, context: INJECTED FAILURE I0720 00:44:12.263533 6138 block_manager_util.cc:139] Instance is unhealthy: IO error: Failed to read metadata file from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/data/block_manager_instance: INJECTED FAILURE: Input/output error (error 5) W0720 00:44:12.272090 5682 consensus_peers.cc:458] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.274837 6138 fs_manager.cc:399] Time spent opening directory manager: real 0.012s user 0.006s sys 0.005s I0720 00:44:12.276247 6138 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:12.276793 6138 file_cache.cc:465] Constructed file cache lbm with capacity 419430 W0720 00:44:12.278249 5682 consensus_peers.cc:458] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:44:12.280690 5558 consensus_peers.cc:458] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0720 00:44:12.289479 5682 consensus_peers.cc:458] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. E0720 00:44:12.292937 6138 log_block_manager.cc:2011] Not using report from /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/data: IO error: Data directory failed (error 5) I0720 00:44:12.293115 6138 fs_manager.cc:419] Time spent opening block manager: real 0.015s user 0.001s sys 0.000s I0720 00:44:12.293181 6138 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" format_stamp: "Formatted at 2019-07-20 00:44:05 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:12.294245 6138 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/wal 2 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-2/data Total live blocks: 35 Total live bytes: 178542 Total live bytes (after alignment): 274432 Total number of LBM containers: 9 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:12.332638 6138 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:12.334605 6138 ts_tablet_manager.cc:345] Loading tablet metadata (0/10 complete) W0720 00:44:12.336979 5682 consensus_peers.cc:458] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.354230 6138 ts_tablet_manager.cc:357] Loaded tablet metadata (10 total tablets, 10 live tablets) I0720 00:44:12.354316 6138 ts_tablet_manager.cc:363] Registering tablets (0/10 complete) I0720 00:44:12.357409 6159 tablet_bootstrap.cc:439] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.357726 6159 ts_tablet_manager.cc:1080] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.357806 6159 ts_tablet_manager.cc:1059] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0720 00:44:12.357908 6159 tablet_replica.cc:273] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.358028 6159 raft_consensus.cc:2147] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.358939 6159 raft_consensus.cc:2174] T dff7af31a11e4cf78f6efbbfa12762d3 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.360791 6159 tablet_bootstrap.cc:439] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.360918 6159 ts_tablet_manager.cc:1080] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.360993 6159 ts_tablet_manager.cc:1059] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.361053 6159 tablet_replica.cc:273] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.361122 6159 raft_consensus.cc:2147] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.361184 6159 raft_consensus.cc:2174] T f0f53e18f4ec4eec823ef551f843b54e P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.362306 6159 tablet_bootstrap.cc:439] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.362422 6159 ts_tablet_manager.cc:1080] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.362510 6159 ts_tablet_manager.cc:1059] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.362571 6159 tablet_replica.cc:273] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.362654 6159 raft_consensus.cc:2147] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.362710 6159 raft_consensus.cc:2174] T d54fc0a6131248cc929ba6d4b0a17489 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.364027 6159 tablet_bootstrap.cc:439] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.364140 6159 ts_tablet_manager.cc:1080] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.364214 6159 ts_tablet_manager.cc:1059] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.364279 6159 tablet_replica.cc:273] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.364342 6159 raft_consensus.cc:2147] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.364409 6159 raft_consensus.cc:2174] T c1e0c42e42524c22af51c8e4a457d743 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.365561 6159 tablet_bootstrap.cc:439] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.365681 6159 ts_tablet_manager.cc:1080] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.365854 6159 ts_tablet_manager.cc:1059] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.365921 6159 tablet_replica.cc:273] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.365983 6159 raft_consensus.cc:2147] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.366048 6159 raft_consensus.cc:2174] T 43e3ab995f34471c92e887c28c2e2e0f P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.367190 6159 tablet_bootstrap.cc:439] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.367324 6159 ts_tablet_manager.cc:1080] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.367404 6159 ts_tablet_manager.cc:1059] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.367457 6159 tablet_replica.cc:273] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.367518 6159 raft_consensus.cc:2147] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.367583 6159 raft_consensus.cc:2174] T 6c0c6e70945746faa55c4964f07a5a86 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.368619 6159 tablet_bootstrap.cc:439] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.368744 6159 ts_tablet_manager.cc:1080] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.368834 6159 ts_tablet_manager.cc:1059] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.368894 6159 tablet_replica.cc:273] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.369045 6159 raft_consensus.cc:2147] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.369112 6159 raft_consensus.cc:2174] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.370143 6159 tablet_bootstrap.cc:439] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.370297 6159 ts_tablet_manager.cc:1080] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.370364 6159 ts_tablet_manager.cc:1059] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.370422 6159 tablet_replica.cc:273] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.370488 6159 raft_consensus.cc:2147] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.370550 6159 raft_consensus.cc:2174] T 736b3f88dd4a41fbb23ef072e4e28905 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.371771 6159 tablet_bootstrap.cc:439] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.371899 6159 ts_tablet_manager.cc:1080] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.372069 6159 ts_tablet_manager.cc:1059] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.372117 6159 tablet_replica.cc:273] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.373286 6159 raft_consensus.cc:2147] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.373450 6159 raft_consensus.cc:2174] T f50d69e061a145869a005de2b35d1acc P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:12.374553 6138 ts_tablet_manager.cc:377] Registered 10 tablets I0720 00:44:12.374703 6159 tablet_bootstrap.cc:439] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Bootstrap starting. E0720 00:44:12.374855 6159 ts_tablet_manager.cc:1080] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Tablet failed to bootstrap: IO error: some tablet data is in a failed directory I0720 00:44:12.374963 6159 ts_tablet_manager.cc:1059] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: Time spent bootstrapping tablet: real 0.000s user 0.000s sys 0.000s I0720 00:44:12.375059 6159 tablet_replica.cc:273] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e: stopping tablet replica I0720 00:44:12.375130 6159 raft_consensus.cc:2147] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:12.375190 6159 raft_consensus.cc:2174] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P be9af1eb9bb14a7e8ef0cf71154d5e6e [term 1 FOLLOWER]: Raft consensus is shut down! W0720 00:44:12.393503 5558 consensus_peers.cc:458] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Status: Network error: Client connection negotiation failed: client connection to 127.0.58.129:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.394718 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000320 I0720 00:44:12.403894 6138 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:34347 I0720 00:44:12.405007 6138 webserver.cc:374] Webserver started at http://127.0.58.129:42415/ using document root and password file I0720 00:44:12.405462 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":4961,"cfile_init":1,"data dirs.queue_time_us":94,"data dirs.run_cpu_time_us":140,"data dirs.run_wall_time_us":605,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"rows_written":13,"thread_start_us":296,"threads_started":1,"wal-append.queue_time_us":440} I0720 00:44:12.405907 6138 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:44:12.406627 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 264 bytes on disk I0720 00:44:12.407392 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:12.408502 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.860168 I0720 00:44:12.415992 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 6138 I0720 00:44:12.420461 6265 heartbeater.cc:324] Connected to a master server at 127.0.58.190:38255 I0720 00:44:12.420701 6265 heartbeater.cc:405] Registering TS with master... I0720 00:44:12.421277 6265 heartbeater.cc:454] Master 127.0.58.190:38255 requested a full tablet report, sending... I0720 00:44:12.421481 234 disk_failure-itest.cc:256] Currently has 10 failed tablets I0720 00:44:12.421525 234 disk_failure-itest.cc:234] Resetting error injection flags I0720 00:44:12.423379 5374 ts_manager.cc:161] Re-registered known tserver with Master: be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) W0720 00:44:12.423934 5374 catalog_manager.cc:3988] Tablet dff7af31a11e4cf78f6efbbfa12762d3 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424067 5374 catalog_manager.cc:3988] Tablet f0f53e18f4ec4eec823ef551f843b54e (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424110 5374 catalog_manager.cc:3988] Tablet d54fc0a6131248cc929ba6d4b0a17489 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424150 5374 catalog_manager.cc:3988] Tablet c1e0c42e42524c22af51c8e4a457d743 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424183 5374 catalog_manager.cc:3988] Tablet 43e3ab995f34471c92e887c28c2e2e0f (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424222 5374 catalog_manager.cc:3988] Tablet 736b3f88dd4a41fbb23ef072e4e28905 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424253 5374 catalog_manager.cc:3988] Tablet 6c0c6e70945746faa55c4964f07a5a86 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424294 5374 catalog_manager.cc:3988] Tablet f50d69e061a145869a005de2b35d1acc (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424325 5374 catalog_manager.cc:3988] Tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:12.424363 5374 catalog_manager.cc:3988] Tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory I0720 00:44:12.425734 5374 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:39287 I0720 00:44:12.433004 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.024s user 0.024s sys 0.000s Metrics: {"bytes_written":17346,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9252,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26716,"cfile_init":5,"data dirs.queue_time_us":1072,"data dirs.run_cpu_time_us":578,"data dirs.run_wall_time_us":1053,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":661,"lbm_reads_lt_1ms":46,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":495} I0720 00:44:12.433926 5794 maintenance_manager.cc:327] P aedbe741f4014e40ae382c213ab0d151: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 4809 bytes on disk W0720 00:44:12.434443 6262 generic_service.cc:128] {username='slave'} at 127.0.0.1:52320 forcing change of non-runtime-safe flag env_inject_eio I0720 00:44:12.434576 6262 generic_service.cc:148] {username='slave'} at 127.0.0.1:52320 changed flags via RPC: env_inject_eio from '1' to '0.0' I0720 00:44:12.434514 5688 maintenance_manager.cc:532] P aedbe741f4014e40ae382c213ab0d151: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} W0720 00:44:12.436069 6262 generic_service.cc:128] {username='slave'} at 127.0.0.1:52320 forcing change of non-runtime-safe flag cfile_inject_corruption I0720 00:44:12.436134 6262 generic_service.cc:148] {username='slave'} at 127.0.0.1:52320 changed flags via RPC: cfile_inject_corruption from '0' to '0.0' W0720 00:44:12.437062 6262 generic_service.cc:128] {username='slave'} at 127.0.0.1:52320 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:12.437120 6262 generic_service.cc:148] {username='slave'} at 127.0.0.1:52320 changed flags via RPC: enable_tablet_copy from 'true' to 'true' I0720 00:44:12.507385 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.000343 Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ad906cc98588459aa9641d8f4ea26ed9 | 127.0.58.190:38255 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 445c7e479f0246a5b26400b1cd4d8d27 | 127.0.58.130:39833 | HEALTHY | aedbe741f4014e40ae382c213ab0d151 | 127.0.58.131:46091 | HEALTHY | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | HEALTHY | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:34347 enable_tablet_copy | false | unsafe | 127.0.58.130:39833, 127.0.58.131:46091, 127.0.58.132:44535 enable_tablet_copy | true | unsafe | 127.0.58.129:34347 env_inject_eio | 0 | hidden | 127.0.58.129:34347 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34347 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34347 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:39833 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:46091 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:44535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34347 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:39833 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:46091 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:44535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 6c0c6e70945746faa55c4964f07a5a86 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet f50d69e061a145869a005de2b35d1acc of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | [config not available] | | | B | A* B C | 1 | -1 | Yes C | A* B C | 1 | -1 | Yes Tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | [config not available] | | | B | A B C* | 1 | -1 | Yes C | A B C* | 1 | -1 | Yes Tablet d54fc0a6131248cc929ba6d4b0a17489 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Tablet 43e3ab995f34471c92e887c28c2e2e0f of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Tablet dff7af31a11e4cf78f6efbbfa12762d3 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Tablet 736b3f88dd4a41fbb23ef072e4e28905 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet c1e0c42e42524c22af51c8e4a457d743 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet f0f53e18f4ec4eec823ef551f843b54e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 0 | 0 | 10 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:12.517575 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: FlushMRSOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.010s user 0.004s sys 0.002s Metrics: {"bytes_written":4695,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":166,"data dirs.run_wall_time_us":674,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"rows_written":2,"thread_start_us":260,"threads_started":1,"wal-append.queue_time_us":355} I0720 00:44:12.518646 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 167 bytes on disk I0720 00:44:12.519373 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:44:12.519294 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:12.519841 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87): perf score=0.406368 I0720 00:44:12.537395 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: CompactRowSetsOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.017s user 0.011s sys 0.004s Metrics: {"bytes_written":17346,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9262,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26666,"cfile_init":5,"data dirs.queue_time_us":670,"data dirs.run_cpu_time_us":346,"data dirs.run_wall_time_us":830,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":599,"lbm_reads_lt_1ms":46,"lbm_write_time_us":497,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":495,"spinlock_wait_cycles":1920} I0720 00:44:12.538318 5670 maintenance_manager.cc:327] P 445c7e479f0246a5b26400b1cd4d8d27: Scheduling UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87): 4809 bytes on disk I0720 00:44:12.538759 5564 maintenance_manager.cc:532] P 445c7e479f0246a5b26400b1cd4d8d27: UndoDeltaBlockGCOp(7dfcac69dd8c4d54a4fe10c404a9cc87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} I0720 00:44:12.663869 5558 consensus_queue.cc:544] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.665802 5374 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet d54fc0a6131248cc929ba6d4b0a17489 (attempt 1) I0720 00:44:12.667023 5682 consensus_queue.cc:544] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.667476 5621 consensus_queue.cc:206] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.669203 5374 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 43e3ab995f34471c92e887c28c2e2e0f (attempt 1) I0720 00:44:12.671097 5745 consensus_queue.cc:206] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.674706 5621 raft_consensus.cc:1184] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.675081 5745 raft_consensus.cc:1184] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:12.675248 6302 consensus_queue.cc:984] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.675668 6300 consensus_queue.cc:984] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:12.677305 5970 raft_consensus.cc:2792] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.677800 5621 raft_consensus.cc:2792] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.684721 5361 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 43e3ab995f34471c92e887c28c2e2e0f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.685775 5991 raft_consensus.cc:2792] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.687091 5374 catalog_manager.cc:4067] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.689203 5362 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet d54fc0a6131248cc929ba6d4b0a17489 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.690250 5748 raft_consensus.cc:2792] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.693389 5374 catalog_manager.cc:4067] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:12.696584 5374 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet d54fc0a6131248cc929ba6d4b0a17489 (attempt 1) I0720 00:44:12.696782 5369 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 43e3ab995f34471c92e887c28c2e2e0f (attempt 1) I0720 00:44:12.698454 5745 consensus_queue.cc:206] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.699273 5369 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 43e3ab995f34471c92e887c28c2e2e0f (attempt 1) I0720 00:44:12.700522 5626 raft_consensus.cc:1184] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:12.701097 5361 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 43e3ab995f34471c92e887c28c2e2e0f with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER failed with leader aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } }. Pending config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.701248 5361 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 43e3ab995f34471c92e887c28c2e2e0f with cas_config_opid_index 106 with a delay of 32 ms (attempt = 1) I0720 00:44:12.701426 5970 consensus_queue.cc:984] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:12.702797 5682 consensus_queue.cc:544] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.704488 5621 consensus_queue.cc:206] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.705174 5369 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (attempt 1) I0720 00:44:12.706388 6302 raft_consensus.cc:2792] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } W0720 00:44:12.708328 5683 consensus_peers.cc:458] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e3ab995f34471c92e887c28c2e2e0f. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.709199 5626 raft_consensus.cc:2792] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.711500 5748 raft_consensus.cc:1184] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.712148 5991 consensus_queue.cc:984] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.713122 5745 consensus_queue.cc:206] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.714810 5991 raft_consensus.cc:2792] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.715190 5369 catalog_manager.cc:4067] T 43e3ab995f34471c92e887c28c2e2e0f P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.717363 5362 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d54fc0a6131248cc929ba6d4b0a17489 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0720 00:44:12.717468 5559 consensus_peers.cc:458] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d54fc0a6131248cc929ba6d4b0a17489. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.717979 5745 raft_consensus.cc:2792] T d54fc0a6131248cc929ba6d4b0a17489 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.718834 5626 raft_consensus.cc:1184] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:12.719398 6314 consensus_queue.cc:984] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:12.721501 5374 catalog_manager.cc:4067] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.722467 5361 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 43e3ab995f34471c92e887c28c2e2e0f with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:12.726161 6313 raft_consensus.cc:2792] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.729395 5361 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.730628 5626 raft_consensus.cc:2792] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.730873 5374 catalog_manager.cc:4067] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:12.733834 5374 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (attempt 1) I0720 00:44:12.734450 5361 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 43e3ab995f34471c92e887c28c2e2e0f with cas_config_opid_index 106: aborting the task: latest config opid_index 107; task opid_index 106 I0720 00:44:12.736575 5374 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (attempt 1) I0720 00:44:12.737018 5745 consensus_queue.cc:206] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.738567 5361 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER failed with leader aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } }. Pending config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.738662 5361 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a with cas_config_opid_index 105 with a delay of 63 ms (attempt = 1) W0720 00:44:12.746455 5683 consensus_peers.cc:458] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26b0b6bdff8a4bc7be9e3eeff65c3b9a. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.746584 5626 raft_consensus.cc:1184] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.747045 6314 consensus_queue.cc:984] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.749219 6302 raft_consensus.cc:2792] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.749804 5626 raft_consensus.cc:2792] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.752323 5361 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:12.753690 5369 catalog_manager.cc:4067] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.774657 5558 consensus_queue.cc:544] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.777184 5376 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet c1e0c42e42524c22af51c8e4a457d743 (attempt 1) I0720 00:44:12.778558 5626 consensus_queue.cc:206] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ad906cc98588459aa9641d8f4ea26ed9 | 127.0.58.190:38255 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary I0720 00:44:12.780723 5745 raft_consensus.cc:1184] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 445c7e479f0246a5b26400b1cd4d8d27 | 127.0.58.130:39833 | HEALTHY | aedbe741f4014e40ae382c213ab0d151 | 127.0.58.131:46091 | HEALTHY | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | HEALTHY | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:34347 enable_tablet_copy | false | unsafe | 127.0.58.130:39833, 127.0.58.131:46091, 127.0.58.132:44535 enable_tablet_copy | true | unsafe | 127.0.58.129:34347 env_inject_eio | 0 | hidden | 127.0.58.129:34347 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34347 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34347 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:39833 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:46091 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:44535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34347 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:39833 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:46091 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:44535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 6c0c6e70945746faa55c4964f07a5a86 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING I0720 00:44:12.781174 6300 consensus_queue.cc:984] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet f50d69e061a145869a005de2b35d1acc of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | [config not available] | | | B | A* B C | 1 | -1 | Yes C | A* B C | 1 | -1 | Yes Tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = aedbe741f4014e40ae382c213ab0d151 C = b2196b47ae0e4ba0b1a776ff7d3e6053 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* | | | Yes A | A B* C~ | 1 | 106 | Yes B | A B* C~ | 1 | 106 | Yes Tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Tablet dff7af31a11e4cf78f6efbbfa12762d3 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Tablet 736b3f88dd4a41fbb23ef072e4e28905 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 1 | -1 | Yes C | A B* C | 1 | -1 | Yes Tablet c1e0c42e42524c22af51c8e4a457d743 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | A* B C | 1 | -1 | Yes B | [config not available] | | | C | A* B C | 1 | -1 | Yes Tablet f0f53e18f4ec4eec823ef551f843b54e of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 I0720 00:44:12.783831 5991 raft_consensus.cc:2792] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C* | 1 | -1 | Yes B | [config not available] | | | C | A B C* | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 10 | 2 | 0 | 7 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 2 First Quartile | 7 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | 2 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 29 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:12.784991 5748 raft_consensus.cc:2792] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.786716 5362 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet c1e0c42e42524c22af51c8e4a457d743 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.787644 5376 catalog_manager.cc:4067] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:12.791399 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:44:12.791644 5558 consensus_queue.cc:544] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.791908 5376 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c1e0c42e42524c22af51c8e4a457d743 (attempt 1) I0720 00:44:12.793038 5626 consensus_queue.cc:206] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } W0720 00:44:12.794857 5559 consensus_peers.cc:458] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1e0c42e42524c22af51c8e4a457d743. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.795322 5376 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 736b3f88dd4a41fbb23ef072e4e28905 (attempt 1) I0720 00:44:12.795828 5745 raft_consensus.cc:1184] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.795961 5682 consensus_queue.cc:544] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.796315 6300 consensus_queue.cc:984] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.796809 5626 consensus_queue.cc:206] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.797914 5376 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet f0f53e18f4ec4eec823ef551f843b54e (attempt 1) I0720 00:44:12.798332 6300 raft_consensus.cc:2792] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.798853 5745 consensus_queue.cc:206] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.800328 5621 raft_consensus.cc:1184] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.800766 6314 consensus_queue.cc:984] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.801288 5748 raft_consensus.cc:2792] T c1e0c42e42524c22af51c8e4a457d743 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.801976 5362 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c1e0c42e42524c22af51c8e4a457d743 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:12.802527 5361 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a with cas_config_opid_index 105: aborting the task: latest config opid_index 106; task opid_index 105 I0720 00:44:12.802527 5376 catalog_manager.cc:4067] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.803014 6313 raft_consensus.cc:2792] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.803489 5621 raft_consensus.cc:2792] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.806156 5361 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet f0f53e18f4ec4eec823ef551f843b54e with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.807127 5745 raft_consensus.cc:1184] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:12.807391 5376 catalog_manager.cc:4067] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:12.807655 6301 consensus_queue.cc:984] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:12.810032 5376 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet f0f53e18f4ec4eec823ef551f843b54e (attempt 1) I0720 00:44:12.810124 6300 raft_consensus.cc:2792] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.810670 5745 raft_consensus.cc:2792] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.811295 5748 consensus_queue.cc:206] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.813336 5362 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 736b3f88dd4a41fbb23ef072e4e28905 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) W0720 00:44:12.813421 5683 consensus_peers.cc:458] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0f53e18f4ec4eec823ef551f843b54e. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.814061 5369 catalog_manager.cc:4067] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:12.814371 5621 raft_consensus.cc:1184] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:12.814818 6302 consensus_queue.cc:984] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:12.816197 5558 consensus_queue.cc:544] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.817018 5970 raft_consensus.cc:2792] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.817492 5369 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 736b3f88dd4a41fbb23ef072e4e28905 (attempt 1) I0720 00:44:12.817457 5621 raft_consensus.cc:2792] T f0f53e18f4ec4eec823ef551f843b54e P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.818964 5626 consensus_queue.cc:206] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.819967 5361 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f0f53e18f4ec4eec823ef551f843b54e with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) W0720 00:44:12.820803 5559 consensus_peers.cc:458] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 736b3f88dd4a41fbb23ef072e4e28905. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.821291 5369 catalog_manager.cc:4067] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.821774 5748 raft_consensus.cc:1184] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.822274 5991 consensus_queue.cc:984] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.822522 5376 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 6c0c6e70945746faa55c4964f07a5a86 (attempt 1) I0720 00:44:12.823455 5626 consensus_queue.cc:206] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.824293 5991 raft_consensus.cc:2792] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.824856 5748 raft_consensus.cc:1184] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.825240 6300 consensus_queue.cc:984] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.825188 5745 raft_consensus.cc:2792] T 736b3f88dd4a41fbb23ef072e4e28905 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.827304 6346 raft_consensus.cc:2792] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.827585 5362 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 736b3f88dd4a41fbb23ef072e4e28905 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:12.827869 5748 raft_consensus.cc:2792] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.828900 5376 catalog_manager.cc:4067] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.830257 5362 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 6c0c6e70945746faa55c4964f07a5a86 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.833161 5369 catalog_manager.cc:4067] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index -1 to 106, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.835750 5376 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 6c0c6e70945746faa55c4964f07a5a86 (attempt 1) I0720 00:44:12.836926 5626 consensus_queue.cc:206] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.838510 5748 raft_consensus.cc:1184] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) W0720 00:44:12.838694 5559 consensus_peers.cc:458] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c0c6e70945746faa55c4964f07a5a86. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.838948 6346 consensus_queue.cc:984] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:12.840723 6300 raft_consensus.cc:2792] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.841323 5748 raft_consensus.cc:2792] T 6c0c6e70945746faa55c4964f07a5a86 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.842988 5362 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6c0c6e70945746faa55c4964f07a5a86 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:12.844444 5376 catalog_manager.cc:4067] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index 106 to 107, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.859175 6128 heartbeater.cc:446] Master 127.0.58.190:38255 was elected leader, sending a full tablet report... I0720 00:44:12.911412 5682 consensus_queue.cc:544] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed I0720 00:44:12.913772 5369 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet dff7af31a11e4cf78f6efbbfa12762d3 (attempt 1) I0720 00:44:12.914845 5748 consensus_queue.cc:206] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:12.916752 5626 raft_consensus.cc:1184] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:12.917271 6313 consensus_queue.cc:984] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:12.919214 6314 raft_consensus.cc:2792] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.919768 5626 raft_consensus.cc:2792] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:12.922179 5361 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet dff7af31a11e4cf78f6efbbfa12762d3 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:12.922957 5369 catalog_manager.cc:4067] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index -1 to 105, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:12.928411 5369 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet dff7af31a11e4cf78f6efbbfa12762d3 (attempt 1) I0720 00:44:12.929706 5748 consensus_queue.cc:206] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.930490 5369 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet dff7af31a11e4cf78f6efbbfa12762d3 (attempt 1) I0720 00:44:12.931717 5626 raft_consensus.cc:1184] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:12.931819 5361 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dff7af31a11e4cf78f6efbbfa12762d3 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER failed with leader aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } }. Pending config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:12.931923 5361 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dff7af31a11e4cf78f6efbbfa12762d3 with cas_config_opid_index 105 with a delay of 64 ms (attempt = 1) W0720 00:44:12.931982 5683 consensus_peers.cc:458] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dff7af31a11e4cf78f6efbbfa12762d3. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:12.932210 6314 consensus_queue.cc:984] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:12.934042 6313 raft_consensus.cc:2792] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.934592 5626 raft_consensus.cc:2792] T dff7af31a11e4cf78f6efbbfa12762d3 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:12.936368 5361 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dff7af31a11e4cf78f6efbbfa12762d3 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:12.937615 5369 catalog_manager.cc:4067] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index 105 to 106, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:12.996948 5361 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dff7af31a11e4cf78f6efbbfa12762d3 with cas_config_opid_index 105: aborting the task: latest config opid_index 106; task opid_index 105 Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ad906cc98588459aa9641d8f4ea26ed9 | 127.0.58.190:38255 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 445c7e479f0246a5b26400b1cd4d8d27 | 127.0.58.130:39833 | HEALTHY | aedbe741f4014e40ae382c213ab0d151 | 127.0.58.131:46091 | HEALTHY | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | HEALTHY | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:34347 enable_tablet_copy | false | unsafe | 127.0.58.130:39833, 127.0.58.131:46091, 127.0.58.132:44535 enable_tablet_copy | true | unsafe | 127.0.58.129:34347 env_inject_eio | 0 | hidden | 127.0.58.129:34347 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34347 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34347 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:39833 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:46091 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:44535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34347 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:39833 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:46091 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:44535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet f50d69e061a145869a005de2b35d1acc of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A* B C | | | Yes A | [config not available] | | | B | A* B C | 1 | -1 | Yes C | A* B C | 1 | -1 | Yes Tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running [LEADER] State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | -1 | Yes B | [config not available] | | | C | A B* C | 1 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 8 | 0 | 2 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 2 First Quartile | 8 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | 2 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:13.082214 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors W0720 00:44:13.092957 6313 consensus_peers.cc:251] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.137794 6346 raft_consensus.cc:472] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be9af1eb9bb14a7e8ef0cf71154d5e6e) I0720 00:44:13.137950 6346 raft_consensus.cc:494] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.138558 6346 leader_election.cc:296] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:13.139333 5748 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:13.139569 5748 raft_consensus.cc:2311] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 445c7e479f0246a5b26400b1cd4d8d27 for term 2 because replica is either leader or believes a valid leader to be alive. I0720 00:44:13.140514 6221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true W0720 00:44:13.143376 5558 leader_election.cc:348] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Illegal state: cannot vote while shut down I0720 00:44:13.143461 5558 leader_election.cc:310] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27; no voters: aedbe741f4014e40ae382c213ab0d151, be9af1eb9bb14a7e8ef0cf71154d5e6e I0720 00:44:13.143757 6346 raft_consensus.cc:2592] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:13.162220 6346 consensus_peers.cc:251] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.203037 6313 raft_consensus.cc:472] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be9af1eb9bb14a7e8ef0cf71154d5e6e) I0720 00:44:13.203222 6313 raft_consensus.cc:494] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.203824 6313 leader_election.cc:296] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833) I0720 00:44:13.204450 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:13.204483 6221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:13.204651 5626 raft_consensus.cc:2311] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aedbe741f4014e40ae382c213ab0d151 for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:13.205272 5682 leader_election.cc:348] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Illegal state: cannot vote while shut down I0720 00:44:13.205354 5682 leader_election.cc:310] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aedbe741f4014e40ae382c213ab0d151; no voters: 445c7e479f0246a5b26400b1cd4d8d27, be9af1eb9bb14a7e8ef0cf71154d5e6e I0720 00:44:13.205626 6313 raft_consensus.cc:2592] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:13.259683 6346 consensus_peers.cc:251] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:13.261405 6313 consensus_peers.cc:251] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.287735 6313 raft_consensus.cc:472] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be9af1eb9bb14a7e8ef0cf71154d5e6e) I0720 00:44:13.287896 6313 raft_consensus.cc:494] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.288450 6313 leader_election.cc:296] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:13.288941 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" is_pre_election: true I0720 00:44:13.289109 5626 raft_consensus.cc:2350] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 1. I0720 00:44:13.289033 6221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:13.289502 5684 leader_election.cc:310] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: W0720 00:44:13.289628 5682 leader_election.cc:348] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Illegal state: cannot vote while shut down I0720 00:44:13.289831 6313 raft_consensus.cc:2644] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:13.289906 6313 raft_consensus.cc:472] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Starting leader election (detected failure of leader be9af1eb9bb14a7e8ef0cf71154d5e6e) I0720 00:44:13.289969 6313 raft_consensus.cc:2890] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:13.291586 6313 raft_consensus.cc:494] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.292032 6313 leader_election.cc:296] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 election: Requested vote from peers 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833), be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347) I0720 00:44:13.292460 5626 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "445c7e479f0246a5b26400b1cd4d8d27" I0720 00:44:13.292526 6221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7dfcac69dd8c4d54a4fe10c404a9cc87" candidate_uuid: "aedbe741f4014e40ae382c213ab0d151" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:13.292608 5626 raft_consensus.cc:2890] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:13.292980 5682 leader_election.cc:348] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Illegal state: cannot vote while shut down I0720 00:44:13.294241 5626 raft_consensus.cc:2350] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aedbe741f4014e40ae382c213ab0d151 in term 2. I0720 00:44:13.294584 5684 leader_election.cc:310] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: be9af1eb9bb14a7e8ef0cf71154d5e6e I0720 00:44:13.294842 6313 raft_consensus.cc:2644] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:13.294941 6313 raft_consensus.cc:667] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 2 LEADER]: Becoming Leader. State: Replica: aedbe741f4014e40ae382c213ab0d151, State: Running, Role: LEADER I0720 00:44:13.295146 6313 consensus_queue.cc:206] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.297098 5376 catalog_manager.cc:4067] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 reported cstate change: term changed from 1 to 2, leader changed from be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) to aedbe741f4014e40ae382c213ab0d151 (127.0.58.131). New cstate: current_term: 2 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:13.361889 6346 raft_consensus.cc:472] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader be9af1eb9bb14a7e8ef0cf71154d5e6e) I0720 00:44:13.362049 6346 raft_consensus.cc:494] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.362704 6346 leader_election.cc:296] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:13.363342 5748 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" is_pre_election: true I0720 00:44:13.363351 6221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" is_pre_election: true I0720 00:44:13.363521 5748 raft_consensus.cc:2350] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 1. W0720 00:44:13.363875 5558 leader_election.cc:348] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Illegal state: cannot vote while shut down I0720 00:44:13.363975 5559 leader_election.cc:310] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: be9af1eb9bb14a7e8ef0cf71154d5e6e I0720 00:44:13.364218 6346 raft_consensus.cc:2644] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:13.364265 6346 raft_consensus.cc:472] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader be9af1eb9bb14a7e8ef0cf71154d5e6e) I0720 00:44:13.364303 6346 raft_consensus.cc:2890] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:13.365751 6372 consensus_peers.cc:251] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.365792 6346 raft_consensus.cc:494] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.366261 6346 leader_election.cc:296] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 election: Requested vote from peers be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347), aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091) I0720 00:44:13.366797 5748 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "aedbe741f4014e40ae382c213ab0d151" I0720 00:44:13.366796 6221 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f50d69e061a145869a005de2b35d1acc" candidate_uuid: "445c7e479f0246a5b26400b1cd4d8d27" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" I0720 00:44:13.366973 5748 raft_consensus.cc:2890] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:13.367276 5558 leader_election.cc:348] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Illegal state: cannot vote while shut down I0720 00:44:13.368574 5748 raft_consensus.cc:2350] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 445c7e479f0246a5b26400b1cd4d8d27 in term 2. I0720 00:44:13.368922 5559 leader_election.cc:310] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 445c7e479f0246a5b26400b1cd4d8d27, aedbe741f4014e40ae382c213ab0d151; no voters: be9af1eb9bb14a7e8ef0cf71154d5e6e I0720 00:44:13.369143 6346 raft_consensus.cc:2644] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:13.369542 6346 raft_consensus.cc:667] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 LEADER]: Becoming Leader. State: Replica: 445c7e479f0246a5b26400b1cd4d8d27, State: Running, Role: LEADER I0720 00:44:13.369740 6346 consensus_queue.cc:206] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.372107 5376 catalog_manager.cc:4067] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: term changed from 1 to 2, leader changed from be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) to 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130). New cstate: current_term: 2 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be9af1eb9bb14a7e8ef0cf71154d5e6e" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 34347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: UNKNOWN } } } W0720 00:44:13.420018 6313 consensus_peers.cc:251] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.429141 6265 heartbeater.cc:446] Master 127.0.58.190:38255 was elected leader, sending a full tablet report... W0720 00:44:13.431002 5376 catalog_manager.cc:3988] Tablet dff7af31a11e4cf78f6efbbfa12762d3 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431094 5376 catalog_manager.cc:3988] Tablet f0f53e18f4ec4eec823ef551f843b54e (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431167 5376 catalog_manager.cc:3988] Tablet d54fc0a6131248cc929ba6d4b0a17489 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431236 5376 catalog_manager.cc:3988] Tablet c1e0c42e42524c22af51c8e4a457d743 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431320 5376 catalog_manager.cc:3988] Tablet 43e3ab995f34471c92e887c28c2e2e0f (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431383 5376 catalog_manager.cc:3988] Tablet 736b3f88dd4a41fbb23ef072e4e28905 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431464 5376 catalog_manager.cc:3988] Tablet 6c0c6e70945746faa55c4964f07a5a86 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431527 5376 catalog_manager.cc:3988] Tablet f50d69e061a145869a005de2b35d1acc (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431597 5376 catalog_manager.cc:3988] Tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.431658 5376 catalog_manager.cc:3988] Tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (table test-workload [id=34d3a1886d964668a2d9020148c19d10]) has failed on TS be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): IO error: some tablet data is in a failed directory W0720 00:44:13.434839 6346 consensus_peers.cc:251] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ad906cc98588459aa9641d8f4ea26ed9 | 127.0.58.190:38255 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 445c7e479f0246a5b26400b1cd4d8d27 | 127.0.58.130:39833 | HEALTHY | aedbe741f4014e40ae382c213ab0d151 | 127.0.58.131:46091 | HEALTHY | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | HEALTHY | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:34347 enable_tablet_copy | false | unsafe | 127.0.58.130:39833, 127.0.58.131:46091, 127.0.58.132:44535 enable_tablet_copy | true | unsafe | 127.0.58.129:34347 env_inject_eio | 0 | hidden | 127.0.58.129:34347 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34347 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34347 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:39833 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:46091 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:44535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34347 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:39833 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:46091 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:44535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet f50d69e061a145869a005de2b35d1acc of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING [LEADER] aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING All reported replicas are: A = be9af1eb9bb14a7e8ef0cf71154d5e6e B = 445c7e479f0246a5b26400b1cd4d8d27 C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 2 | -1 | Yes C | A B C | 2 | -1 | Yes Tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): RUNNING be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: some tablet data is in a failed directory aedbe741f4014e40ae382c213ab0d151 (127.0.58.131:46091): RUNNING [LEADER] All reported replicas are: A = 445c7e479f0246a5b26400b1cd4d8d27 B = be9af1eb9bb14a7e8ef0cf71154d5e6e C = aedbe741f4014e40ae382c213ab0d151 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A B C | 2 | -1 | Yes B | [config not available] | | | C | A B C* | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 8 | 0 | 2 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 2 First Quartile | 8 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | 2 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:13.488580 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors W0720 00:44:13.510232 6313 consensus_peers.cc:251] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:13.542703 6313 consensus_peers.cc:251] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:13.647435 6346 consensus_peers.cc:251] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:13.752703 6313 consensus_peers.cc:251] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:13.797003 6346 consensus_peers.cc:251] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.834403 5626 raft_consensus.cc:1184] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:13.835001 6313 consensus_queue.cc:984] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:13.840407 5682 consensus_queue.cc:544] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed W0720 00:44:13.840526 5682 consensus_peers.cc:458] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. W0720 00:44:13.842041 6402 consensus_peers.cc:251] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.843319 5558 consensus_queue.cc:544] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: The tablet replica hosted on peer be9af1eb9bb14a7e8ef0cf71154d5e6e has failed W0720 00:44:13.843443 5558 consensus_peers.cc:458] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129:34347): Couldn't send request to peer be9af1eb9bb14a7e8ef0cf71154d5e6e. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: some tablet data is in a failed directory. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:13.843806 5376 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (attempt 1) I0720 00:44:13.844918 5748 consensus_queue.cc:206] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.846222 5626 raft_consensus.cc:1184] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:13.846660 6402 consensus_queue.cc:984] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:13.848428 6313 raft_consensus.cc:2792] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:13.848944 5626 raft_consensus.cc:2792] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:13.850697 5361 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:13.851763 5376 catalog_manager.cc:4067] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index -1 to 107, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:13.854557 5376 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 (attempt 1) I0720 00:44:13.856053 5748 consensus_queue.cc:206] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:13.857527 5626 raft_consensus.cc:1184] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Refusing update from remote peer aedbe741f4014e40ae382c213ab0d151: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) W0720 00:44:13.857947 5683 consensus_peers.cc:458] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dfcac69dd8c4d54a4fe10c404a9cc87. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:13.858024 6400 consensus_queue.cc:984] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:13.859907 6402 raft_consensus.cc:2792] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:13.860412 5626 raft_consensus.cc:2792] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:13.862131 5361 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:13.863663 5369 catalog_manager.cc:4067] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index 107 to 108, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "aedbe741f4014e40ae382c213ab0d151" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } W0720 00:44:13.873219 6373 consensus_peers.cc:251] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:13.917088 6313 consensus_peers.cc:251] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.919087 5748 raft_consensus.cc:1184] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:13.919631 6373 consensus_queue.cc:984] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:13.924667 5374 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet f50d69e061a145869a005de2b35d1acc (attempt 1) W0720 00:44:13.924873 6373 consensus_peers.cc:251] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:13.925738 5626 consensus_queue.cc:206] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } I0720 00:44:13.927008 5748 raft_consensus.cc:1184] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:13.927460 6373 consensus_queue.cc:984] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:13.929347 6346 raft_consensus.cc:2792] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:13.929919 5748 raft_consensus.cc:2792] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } } I0720 00:44:13.931733 5362 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet f50d69e061a145869a005de2b35d1acc with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:13.932862 5374 catalog_manager.cc:4067] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 reported cstate change: config changed from index -1 to 107, VOTER be9af1eb9bb14a7e8ef0cf71154d5e6e (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } health_report { overall_health: HEALTHY } } } I0720 00:44:13.941066 5374 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet f50d69e061a145869a005de2b35d1acc (attempt 1) I0720 00:44:13.942441 5626 consensus_queue.cc:206] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } I0720 00:44:13.944033 5748 raft_consensus.cc:1184] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Refusing update from remote peer 445c7e479f0246a5b26400b1cd4d8d27: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) W0720 00:44:13.944326 5559 consensus_peers.cc:458] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Couldn't send request to peer b2196b47ae0e4ba0b1a776ff7d3e6053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f50d69e061a145869a005de2b35d1acc. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:13.944568 6346 consensus_queue.cc:984] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:13.946542 6373 raft_consensus.cc:2792] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:13.947144 5748 raft_consensus.cc:2792] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } I0720 00:44:13.949390 5362 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f50d69e061a145869a005de2b35d1acc with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:13.950917 5369 catalog_manager.cc:4067] T f50d69e061a145869a005de2b35d1acc P aedbe741f4014e40ae382c213ab0d151 reported cstate change: config changed from index 107 to 108, NON_VOTER b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "445c7e479f0246a5b26400b1cd4d8d27" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "445c7e479f0246a5b26400b1cd4d8d27" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 39833 } } peers { permanent_uuid: "aedbe741f4014e40ae382c213ab0d151" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 46091 } } peers { permanent_uuid: "b2196b47ae0e4ba0b1a776ff7d3e6053" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 44535 } attrs { promote: true } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- ad906cc98588459aa9641d8f4ea26ed9 | 127.0.58.190:38255 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 445c7e479f0246a5b26400b1cd4d8d27 | 127.0.58.130:39833 | HEALTHY | aedbe741f4014e40ae382c213ab0d151 | 127.0.58.131:46091 | HEALTHY | b2196b47ae0e4ba0b1a776ff7d3e6053 | 127.0.58.132:44535 | HEALTHY | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:34347 enable_tablet_copy | false | unsafe | 127.0.58.130:39833, 127.0.58.131:46091, 127.0.58.132:44535 enable_tablet_copy | true | unsafe | 127.0.58.129:34347 env_inject_eio | 0 | hidden | 127.0.58.129:34347 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:34347 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:34347 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:39833 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:46091 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:44535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:34347 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:39833 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:46091 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailOnBootstrap_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:44535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 10 | 10 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | be9af1eb9bb14a7e8ef0cf71154d5e6e | 127.0.58.129:34347 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:44:14.070686 234 log_verifier.cc:121] Checking tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a W0720 00:44:14.108201 6313 consensus_peers.cc:251] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.137559 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet 26b0b6bdff8a4bc7be9e3eeff65c3b9a I0720 00:44:14.137732 234 log_verifier.cc:121] Checking tablet 43e3ab995f34471c92e887c28c2e2e0f W0720 00:44:14.181360 6346 consensus_peers.cc:251] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.203912 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 43e3ab995f34471c92e887c28c2e2e0f I0720 00:44:14.204107 234 log_verifier.cc:121] Checking tablet 6c0c6e70945746faa55c4964f07a5a86 W0720 00:44:14.253037 6346 consensus_peers.cc:251] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.272689 6313 consensus_peers.cc:251] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.273461 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 6c0c6e70945746faa55c4964f07a5a86 I0720 00:44:14.273619 234 log_verifier.cc:121] Checking tablet 736b3f88dd4a41fbb23ef072e4e28905 W0720 00:44:14.309342 6313 consensus_peers.cc:251] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.324694 6313 consensus_peers.cc:251] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.342917 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet 736b3f88dd4a41fbb23ef072e4e28905 I0720 00:44:14.343117 234 log_verifier.cc:121] Checking tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 W0720 00:44:14.350816 6346 consensus_peers.cc:251] T c1e0c42e42524c22af51c8e4a457d743 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.356093 6346 consensus_peers.cc:251] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.411360 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet 7dfcac69dd8c4d54a4fe10c404a9cc87 I0720 00:44:14.411561 234 log_verifier.cc:121] Checking tablet c1e0c42e42524c22af51c8e4a457d743 W0720 00:44:14.432566 6313 consensus_peers.cc:251] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.435602 6346 consensus_peers.cc:251] T f50d69e061a145869a005de2b35d1acc P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.499174 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet c1e0c42e42524c22af51c8e4a457d743 I0720 00:44:14.499305 234 log_verifier.cc:121] Checking tablet d54fc0a6131248cc929ba6d4b0a17489 I0720 00:44:14.563696 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet d54fc0a6131248cc929ba6d4b0a17489 I0720 00:44:14.563834 234 log_verifier.cc:121] Checking tablet dff7af31a11e4cf78f6efbbfa12762d3 W0720 00:44:14.592005 6313 consensus_peers.cc:251] T 43e3ab995f34471c92e887c28c2e2e0f P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.635483 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet dff7af31a11e4cf78f6efbbfa12762d3 I0720 00:44:14.635617 234 log_verifier.cc:121] Checking tablet f0f53e18f4ec4eec823ef551f843b54e W0720 00:44:14.686190 6313 consensus_peers.cc:251] T dff7af31a11e4cf78f6efbbfa12762d3 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.709296 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet f0f53e18f4ec4eec823ef551f843b54e I0720 00:44:14.709429 234 log_verifier.cc:121] Checking tablet f50d69e061a145869a005de2b35d1acc W0720 00:44:14.733001 6313 consensus_peers.cc:251] T 26b0b6bdff8a4bc7be9e3eeff65c3b9a P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.748602 6346 consensus_peers.cc:251] T d54fc0a6131248cc929ba6d4b0a17489 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.748703 6313 consensus_peers.cc:251] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.785893 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet f50d69e061a145869a005de2b35d1acc I0720 00:44:14.786167 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 6138 I0720 00:44:14.787117 6262 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:14.788157 6346 consensus_peers.cc:251] T 6c0c6e70945746faa55c4964f07a5a86 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:14.826136 6346 consensus_peers.cc:251] T 736b3f88dd4a41fbb23ef072e4e28905 P 445c7e479f0246a5b26400b1cd4d8d27 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:14.840425 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 6138 I0720 00:44:14.851208 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 5548 I0720 00:44:14.852031 5666 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:14.924155 6313 consensus_peers.cc:251] T 7dfcac69dd8c4d54a4fe10c404a9cc87 P aedbe741f4014e40ae382c213ab0d151 -> Peer b2196b47ae0e4ba0b1a776ff7d3e6053 (127.0.58.132:44535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:15.010340 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 5548 I0720 00:44:15.035358 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 5672 W0720 00:44:15.035552 5684 connection.cc:655] client connection to 127.0.58.130:39833 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0720 00:44:15.035689 5684 consensus_peers.cc:458] T f0f53e18f4ec4eec823ef551f843b54e P aedbe741f4014e40ae382c213ab0d151 -> Peer 445c7e479f0246a5b26400b1cd4d8d27 (127.0.58.130:39833): Couldn't send request to peer 445c7e479f0246a5b26400b1cd4d8d27. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I0720 00:44:15.036293 5790 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:15.185838 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 5672 I0720 00:44:15.209059 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 6005 I0720 00:44:15.209973 6125 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:15.261138 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 6005 I0720 00:44:15.270320 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 5350 I0720 00:44:15.271251 5408 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:15.355989 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 5350 [ OK ] DiskError/DiskErrorITest.TestFailOnBootstrap/1 (10220 ms) [ RUN ] DiskError/DiskErrorITest.TestFailDuringScanWorkload/0 I0720 00:44:15.405206 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-2 --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:37529 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_tombstone_evicted_tablet_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:15.599061 6442 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:15.599264 6442 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:15.601759 6442 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:44:15.601800 6442 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:15.601830 6442 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:44:15.601855 6442 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:44:15.606532 6442 master_main.cc:98] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/wal --master_tombstone_evicted_tablet_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:37529 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:15.613075 6442 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 89016us I0720 00:44:15.614311 6442 fs_manager.cc:263] Metadata directory not provided I0720 00:44:15.614393 6442 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:15.614729 6442 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:15.614804 6442 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:15.617378 6442 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "62dd01b770a74740ac39ccdea33401d8" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.617903 6442 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "62dd01b770a74740ac39ccdea33401d8" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.618264 6442 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "62dd01b770a74740ac39ccdea33401d8" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.618589 6442 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "62dd01b770a74740ac39ccdea33401d8" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.627291 6442 fs_manager.cc:503] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s I0720 00:44:15.628646 6442 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:15.629117 6442 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:15.632326 6442 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0720 00:44:15.632382 6442 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/wal uuid: "62dd01b770a74740ac39ccdea33401d8" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.632508 6442 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:15.681912 6442 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:15.695591 6442 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:37529 I0720 00:44:15.696658 6442 webserver.cc:374] Webserver started at http://127.0.58.190:46345/ using document root and password file I0720 00:44:15.697311 6442 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:44:15.698397 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 6442 I0720 00:44:15.698999 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37529 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:15.715044 6503 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8: Bootstrap starting. I0720 00:44:15.717593 6503 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8: No blocks or log segments found. Creating new log. I0720 00:44:15.722615 6503 log.cc:525] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8: Log is configured to *not* fsync() on all Append() calls I0720 00:44:15.727283 6503 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8: No bootstrap required, opened a new log I0720 00:44:15.733633 6503 raft_consensus.cc:340] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } I0720 00:44:15.733866 6503 raft_consensus.cc:366] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:15.733943 6503 raft_consensus.cc:705] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dd01b770a74740ac39ccdea33401d8, State: Initialized, Role: FOLLOWER I0720 00:44:15.734478 6503 consensus_queue.cc:229] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } I0720 00:44:15.734675 6503 raft_consensus.cc:380] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:44:15.734742 6503 raft_consensus.cc:472] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:44:15.734850 6503 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:15.736430 6503 raft_consensus.cc:494] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } I0720 00:44:15.736848 6503 leader_election.cc:310] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62dd01b770a74740ac39ccdea33401d8; no voters: I0720 00:44:15.737516 6503 leader_election.cc:296] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:44:15.737707 6506 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:15.738494 6506 raft_consensus.cc:667] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [term 1 LEADER]: Becoming Leader. State: Replica: 62dd01b770a74740ac39ccdea33401d8, State: Running, Role: LEADER I0720 00:44:15.738926 6506 consensus_queue.cc:206] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } I0720 00:44:15.741829 6503 sys_catalog.cc:340] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "62dd01b770a74740ac39ccdea33401d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } } I0720 00:44:15.742031 6503 sys_catalog.cc:343] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: This master's current role is: LEADER I0720 00:44:15.741874 6507 sys_catalog.cc:340] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62dd01b770a74740ac39ccdea33401d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } } I0720 00:44:15.742300 6507 sys_catalog.cc:343] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: This master's current role is: LEADER I0720 00:44:15.742144 6508 sys_catalog.cc:340] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62dd01b770a74740ac39ccdea33401d8. Latest consensus state: current_term: 1 leader_uuid: "62dd01b770a74740ac39ccdea33401d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62dd01b770a74740ac39ccdea33401d8" member_type: VOTER } } I0720 00:44:15.743130 6508 sys_catalog.cc:343] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: This master's current role is: LEADER I0720 00:44:15.744083 6503 sys_catalog.cc:428] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8 [sys.catalog]: configured and running, proceeding with master startup. I0720 00:44:15.744206 6510 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:44:15.744642 6512 catalog_manager.cc:547] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0720 00:44:15.748442 6510 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:44:15.828495 6510 catalog_manager.cc:959] Generated new certificate authority record I0720 00:44:15.830186 6510 catalog_manager.cc:1084] Loading token signing keys... I0720 00:44:15.842200 6510 catalog_manager.cc:4371] T 00000000000000000000000000000000 P 62dd01b770a74740ac39ccdea33401d8: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:15.868669 6504 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:15.868918 6504 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:15.871062 6504 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:15.871163 6504 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:15.871207 6504 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:15.871286 6504 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:44:15.886260 6504 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:37529 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:15.894242 6504 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 89016us I0720 00:44:15.894934 6504 fs_manager.cc:263] Metadata directory not provided I0720 00:44:15.895020 6504 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:15.895328 6504 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:15.895404 6504 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:15.897646 6504 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "b88011e9e0d24f79882735c95c8fc275" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.898277 6504 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "b88011e9e0d24f79882735c95c8fc275" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.898635 6504 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "b88011e9e0d24f79882735c95c8fc275" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.898960 6504 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "b88011e9e0d24f79882735c95c8fc275" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.919472 6504 fs_manager.cc:503] Time spent creating directory manager: real 0.020s user 0.004s sys 0.016s I0720 00:44:15.920912 6504 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:15.921530 6504 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:15.928076 6504 fs_manager.cc:419] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s I0720 00:44:15.928231 6504 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/wal uuid: "b88011e9e0d24f79882735c95c8fc275" format_stamp: "Formatted at 2019-07-20 00:44:15 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:15.928421 6504 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:15.975296 6504 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:15.977519 6504 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:15.977592 6504 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:16.058035 6504 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:41519 I0720 00:44:16.062321 6504 webserver.cc:374] Webserver started at http://127.0.58.129:44351/ using document root and password file I0720 00:44:16.063009 6504 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:44:16.070611 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 6504 I0720 00:44:16.071341 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37529 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:16.083423 6635 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37529 I0720 00:44:16.083642 6635 heartbeater.cc:405] Registering TS with master... I0720 00:44:16.084115 6635 heartbeater.cc:454] Master 127.0.58.190:37529 requested a full tablet report, sending... I0720 00:44:16.086122 6470 ts_manager.cc:161] Registered new tserver with Master: b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519) I0720 00:44:16.088119 6470 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:52513 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:16.235612 6639 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:16.235824 6639 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:16.237857 6639 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:16.237936 6639 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:16.237970 6639 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:16.238023 6639 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:44:16.241752 6639 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:37529 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:16.249179 6639 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 89516us I0720 00:44:16.249816 6639 fs_manager.cc:263] Metadata directory not provided I0720 00:44:16.249882 6639 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:16.250196 6639 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:16.250272 6639 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:16.252414 6639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "29cb3a99090f4ceeb99efc07786b6426" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.252934 6639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "29cb3a99090f4ceeb99efc07786b6426" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.253238 6639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "29cb3a99090f4ceeb99efc07786b6426" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.253532 6639 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "29cb3a99090f4ceeb99efc07786b6426" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.261921 6639 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.008s sys 0.001s I0720 00:44:16.263234 6639 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:16.263721 6639 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:16.267122 6639 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0720 00:44:16.267192 6639 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/wal uuid: "29cb3a99090f4ceeb99efc07786b6426" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.267360 6639 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:16.306388 6639 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:16.308007 6639 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:16.308065 6639 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:16.331986 6639 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:42329 I0720 00:44:16.333010 6639 webserver.cc:374] Webserver started at http://127.0.58.130:45905/ using document root and password file I0720 00:44:16.333636 6639 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:44:16.340216 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 6639 I0720 00:44:16.341078 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37529 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:16.353240 6760 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37529 I0720 00:44:16.353477 6760 heartbeater.cc:405] Registering TS with master... I0720 00:44:16.353982 6760 heartbeater.cc:454] Master 127.0.58.190:37529 requested a full tablet report, sending... I0720 00:44:16.355161 6470 ts_manager.cc:161] Registered new tserver with Master: 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) I0720 00:44:16.355983 6470 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:58213 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:16.499702 6763 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:16.499928 6763 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:16.501996 6763 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:16.502146 6763 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:16.502193 6763 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:16.502261 6763 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:44:16.505942 6763 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:37529 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:16.513373 6763 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 89516us I0720 00:44:16.514021 6763 fs_manager.cc:263] Metadata directory not provided I0720 00:44:16.514111 6763 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:16.514405 6763 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:16.514472 6763 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:16.516630 6763 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.517179 6763 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.517522 6763 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.517837 6763 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.525740 6763 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.009s sys 0.002s I0720 00:44:16.527024 6763 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:16.527477 6763 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:16.530802 6763 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:44:16.530861 6763 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/wal uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" format_stamp: "Formatted at 2019-07-20 00:44:16 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:16.531018 6763 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:16.575256 6763 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:16.576802 6763 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:16.576855 6763 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:16.598951 6763 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:38005 I0720 00:44:16.599951 6763 webserver.cc:374] Webserver started at http://127.0.58.131:40141/ using document root and password file I0720 00:44:16.600546 6763 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:44:16.606983 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 6763 I0720 00:44:16.615334 6884 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37529 I0720 00:44:16.615537 6884 heartbeater.cc:405] Registering TS with master... I0720 00:44:16.615975 6884 heartbeater.cc:454] Master 127.0.58.190:37529 requested a full tablet report, sending... I0720 00:44:16.617069 6470 ts_manager.cc:161] Registered new tserver with Master: 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.617885 6470 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:55827 I0720 00:44:16.619588 234 external_mini_cluster.cc:628] 3 TS(s) registered with all masters I0720 00:44:16.628813 234 test_util.cc:212] Using random seed: 612519995 I0720 00:44:16.642060 6470 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51922: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s" } partition_schema { range_schema { columns { name: "key" } } } W0720 00:44:16.643636 6470 catalog_manager.cc:1583] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0720 00:44:16.672480 6567 tablet_service.cc:1022] Processing CreateTablet for tablet 49ad7f72eaa04a14bc64fa4f6cbce73c (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:16.672134 6572 tablet_service.cc:1022] Processing CreateTablet for tablet f49376a672014e499d7fc53f9325c6f6 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:16.676307 6570 tablet_service.cc:1022] Processing CreateTablet for tablet 57d8956326d345eba0ced816663169f9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:16.676424 6819 tablet_service.cc:1022] Processing CreateTablet for tablet 57d8956326d345eba0ced816663169f9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:16.676563 6569 tablet_service.cc:1022] Processing CreateTablet for tablet 7139dc2492b147989250601c720f324c (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:16.677099 6568 tablet_service.cc:1022] Processing CreateTablet for tablet 52408e87c8e54a7394a12dc191a9b4a4 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:16.678153 6693 tablet_service.cc:1022] Processing CreateTablet for tablet 52408e87c8e54a7394a12dc191a9b4a4 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:16.678215 6571 tablet_service.cc:1022] Processing CreateTablet for tablet a5154a3c09a5435fb2ac44744e8e7007 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:16.678412 6818 tablet_service.cc:1022] Processing CreateTablet for tablet 7139dc2492b147989250601c720f324c (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:16.675806 6696 tablet_service.cc:1022] Processing CreateTablet for tablet a5154a3c09a5435fb2ac44744e8e7007 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:16.679365 6690 tablet_service.cc:1022] Processing CreateTablet for tablet d4520f8364cb4356bf859651b4a349b2 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:16.678745 6564 tablet_service.cc:1022] Processing CreateTablet for tablet 27478acfc8ee4027a84efa2c19ee11e9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:16.679814 6688 tablet_service.cc:1022] Processing CreateTablet for tablet bf4081672af8427b9d5d11d8dfeae031 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:16.678551 6694 tablet_service.cc:1022] Processing CreateTablet for tablet 7139dc2492b147989250601c720f324c (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:16.680131 6691 tablet_service.cc:1022] Processing CreateTablet for tablet 79f36e06977e420f94ab39786832ecc9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:16.678609 6563 tablet_service.cc:1022] Processing CreateTablet for tablet bf4081672af8427b9d5d11d8dfeae031 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:16.679759 6566 tablet_service.cc:1022] Processing CreateTablet for tablet 79f36e06977e420f94ab39786832ecc9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:16.678966 6565 tablet_service.cc:1022] Processing CreateTablet for tablet d4520f8364cb4356bf859651b4a349b2 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:16.682432 6817 tablet_service.cc:1022] Processing CreateTablet for tablet 52408e87c8e54a7394a12dc191a9b4a4 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:16.683442 6816 tablet_service.cc:1022] Processing CreateTablet for tablet 49ad7f72eaa04a14bc64fa4f6cbce73c (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:16.684473 6815 tablet_service.cc:1022] Processing CreateTablet for tablet 79f36e06977e420f94ab39786832ecc9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:16.685458 6814 tablet_service.cc:1022] Processing CreateTablet for tablet d4520f8364cb4356bf859651b4a349b2 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:16.678620 6697 tablet_service.cc:1022] Processing CreateTablet for tablet f49376a672014e499d7fc53f9325c6f6 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:16.686400 6812 tablet_service.cc:1022] Processing CreateTablet for tablet bf4081672af8427b9d5d11d8dfeae031 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:16.677619 6692 tablet_service.cc:1022] Processing CreateTablet for tablet 49ad7f72eaa04a14bc64fa4f6cbce73c (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:16.687206 6689 tablet_service.cc:1022] Processing CreateTablet for tablet 27478acfc8ee4027a84efa2c19ee11e9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:16.687481 6695 tablet_service.cc:1022] Processing CreateTablet for tablet 57d8956326d345eba0ced816663169f9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:16.687608 6813 tablet_service.cc:1022] Processing CreateTablet for tablet 27478acfc8ee4027a84efa2c19ee11e9 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:16.688407 6821 tablet_service.cc:1022] Processing CreateTablet for tablet f49376a672014e499d7fc53f9325c6f6 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:16.675556 6820 tablet_service.cc:1022] Processing CreateTablet for tablet a5154a3c09a5435fb2ac44744e8e7007 (table=test-workload [id=5577622f8e4e4037809dcc8abdd63b11]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:16.774693 6910 tablet_bootstrap.cc:439] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.774916 6916 tablet_bootstrap.cc:439] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.777400 6910 tablet_bootstrap.cc:590] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.777396 6916 tablet_bootstrap.cc:590] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.778451 6911 tablet_bootstrap.cc:439] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.779954 6911 tablet_bootstrap.cc:590] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.780781 6912 tablet_bootstrap.cc:439] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.783310 6914 tablet_bootstrap.cc:439] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.784577 6909 tablet_bootstrap.cc:439] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.790218 6908 tablet_bootstrap.cc:439] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.792913 6908 tablet_bootstrap.cc:590] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.795133 6915 tablet_bootstrap.cc:439] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.796738 6915 tablet_bootstrap.cc:590] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.797468 6915 log.cc:525] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: Log is configured to *not* fsync() on all Append() calls I0720 00:44:16.798209 6912 tablet_bootstrap.cc:590] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.799903 6916 log.cc:525] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log is configured to *not* fsync() on all Append() calls I0720 00:44:16.799973 6909 tablet_bootstrap.cc:590] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.800031 6913 tablet_bootstrap.cc:439] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.800762 6914 tablet_bootstrap.cc:590] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.801086 6910 log.cc:525] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426: Log is configured to *not* fsync() on all Append() calls I0720 00:44:16.802709 6909 tablet_bootstrap.cc:439] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.802862 6915 tablet_bootstrap.cc:439] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.803297 6913 tablet_bootstrap.cc:590] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.803602 6909 ts_tablet_manager.cc:1059] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s I0720 00:44:16.804296 6916 tablet_bootstrap.cc:439] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.804682 6910 tablet_bootstrap.cc:439] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.804890 6916 ts_tablet_manager.cc:1059] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s I0720 00:44:16.805029 6910 ts_tablet_manager.cc:1059] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s I0720 00:44:16.806463 6912 tablet_bootstrap.cc:439] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.806587 6912 ts_tablet_manager.cc:1059] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s I0720 00:44:16.806589 6908 tablet_bootstrap.cc:439] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.806704 6908 ts_tablet_manager.cc:1059] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s I0720 00:44:16.806892 6911 tablet_bootstrap.cc:439] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.807044 6911 ts_tablet_manager.cc:1059] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s I0720 00:44:16.807075 6914 tablet_bootstrap.cc:439] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.807191 6914 ts_tablet_manager.cc:1059] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s I0720 00:44:16.807351 6915 ts_tablet_manager.cc:1059] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s I0720 00:44:16.808593 6913 tablet_bootstrap.cc:439] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.808724 6913 ts_tablet_manager.cc:1059] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s I0720 00:44:16.820312 6912 raft_consensus.cc:340] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.820958 6912 raft_consensus.cc:366] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.821032 6912 raft_consensus.cc:705] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.820807 6910 raft_consensus.cc:340] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.821260 6910 raft_consensus.cc:366] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.821355 6910 raft_consensus.cc:705] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.820482 6911 raft_consensus.cc:340] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.821854 6911 raft_consensus.cc:366] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.821732 6912 consensus_queue.cc:229] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.821923 6910 consensus_queue.cc:229] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.822129 6911 raft_consensus.cc:705] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.822969 6911 consensus_queue.cc:229] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.823660 6760 heartbeater.cc:446] Master 127.0.58.190:37529 was elected leader, sending a full tablet report... I0720 00:44:16.824549 6916 raft_consensus.cc:340] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.825017 6916 raft_consensus.cc:366] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.825071 6916 raft_consensus.cc:705] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.825584 6909 raft_consensus.cc:340] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.825958 6909 raft_consensus.cc:366] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.825863 6913 raft_consensus.cc:340] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.826022 6909 raft_consensus.cc:705] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.826108 6913 raft_consensus.cc:366] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.826162 6913 raft_consensus.cc:705] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.826243 6908 raft_consensus.cc:340] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.826431 6908 raft_consensus.cc:366] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.826478 6908 raft_consensus.cc:705] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.826663 6913 consensus_queue.cc:229] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.826921 6908 consensus_queue.cc:229] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.827476 6915 raft_consensus.cc:340] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.827687 6915 raft_consensus.cc:366] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.827734 6915 raft_consensus.cc:705] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.827914 6912 ts_tablet_manager.cc:1087] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.021s user 0.006s sys 0.010s I0720 00:44:16.827970 6915 consensus_queue.cc:229] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.828397 6912 tablet_bootstrap.cc:439] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.828719 6635 heartbeater.cc:446] Master 127.0.58.190:37529 was elected leader, sending a full tablet report... I0720 00:44:16.830485 6915 ts_tablet_manager.cc:1087] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.023s user 0.000s sys 0.006s I0720 00:44:16.830425 6914 raft_consensus.cc:340] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.830736 6914 raft_consensus.cc:366] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.830806 6915 tablet_bootstrap.cc:439] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.830826 6914 raft_consensus.cc:705] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.831091 6914 consensus_queue.cc:229] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.831990 6884 heartbeater.cc:446] Master 127.0.58.190:37529 was elected leader, sending a full tablet report... I0720 00:44:16.832281 6915 tablet_bootstrap.cc:590] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.830509 6912 tablet_bootstrap.cc:590] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.833366 6914 ts_tablet_manager.cc:1087] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s I0720 00:44:16.833678 6914 tablet_bootstrap.cc:439] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.833919 6909 consensus_queue.cc:229] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.834995 6914 tablet_bootstrap.cc:590] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. W0720 00:44:16.834993 6761 tablet.cc:1799] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:16.835868 6913 ts_tablet_manager.cc:1087] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.027s user 0.002s sys 0.005s I0720 00:44:16.836096 6913 tablet_bootstrap.cc:439] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.836139 6910 ts_tablet_manager.cc:1087] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.031s user 0.016s sys 0.004s I0720 00:44:16.837116 6913 tablet_bootstrap.cc:590] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.837662 6910 tablet_bootstrap.cc:439] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.838327 6909 ts_tablet_manager.cc:1087] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s I0720 00:44:16.838281 6916 consensus_queue.cc:229] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.838621 6909 tablet_bootstrap.cc:439] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.839159 6910 tablet_bootstrap.cc:590] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.840056 6915 tablet_bootstrap.cc:439] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.839017 6914 tablet_bootstrap.cc:439] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.840495 6914 ts_tablet_manager.cc:1059] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s I0720 00:44:16.840826 6915 ts_tablet_manager.cc:1059] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s I0720 00:44:16.841164 6914 raft_consensus.cc:340] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.841316 6914 raft_consensus.cc:366] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.841614 6916 ts_tablet_manager.cc:1087] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.037s user 0.009s sys 0.000s I0720 00:44:16.841897 6916 tablet_bootstrap.cc:439] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.842304 6913 tablet_bootstrap.cc:439] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.841662 6914 raft_consensus.cc:705] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.842588 6911 ts_tablet_manager.cc:1087] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.035s user 0.000s sys 0.007s I0720 00:44:16.842496 6913 ts_tablet_manager.cc:1059] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s I0720 00:44:16.842806 6911 tablet_bootstrap.cc:439] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.843098 6914 consensus_queue.cc:229] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.843583 6916 tablet_bootstrap.cc:590] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.845000 6908 ts_tablet_manager.cc:1087] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.038s user 0.001s sys 0.007s I0720 00:44:16.845131 6916 tablet_bootstrap.cc:439] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.845261 6908 tablet_bootstrap.cc:439] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.845297 6916 ts_tablet_manager.cc:1059] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s I0720 00:44:16.845325 6911 tablet_bootstrap.cc:590] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.845963 6916 raft_consensus.cc:340] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.846204 6916 raft_consensus.cc:366] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.846305 6916 raft_consensus.cc:705] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.846582 6908 tablet_bootstrap.cc:590] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.846504 6916 consensus_queue.cc:229] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.847007 6916 ts_tablet_manager.cc:1087] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:16.846005 6912 tablet_bootstrap.cc:439] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.847270 6916 tablet_bootstrap.cc:439] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.847450 6912 ts_tablet_manager.cc:1059] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s I0720 00:44:16.840700 6909 tablet_bootstrap.cc:590] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.848157 6912 raft_consensus.cc:340] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.846619 6910 tablet_bootstrap.cc:439] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.848500 6916 tablet_bootstrap.cc:590] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.848502 6910 ts_tablet_manager.cc:1059] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s I0720 00:44:16.848944 6908 tablet_bootstrap.cc:439] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.849113 6908 ts_tablet_manager.cc:1059] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I0720 00:44:16.849187 6910 raft_consensus.cc:340] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.849406 6910 raft_consensus.cc:366] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.849494 6910 raft_consensus.cc:705] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.849750 6910 consensus_queue.cc:229] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.849774 6908 raft_consensus.cc:340] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.849939 6914 ts_tablet_manager.cc:1087] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s I0720 00:44:16.849962 6908 raft_consensus.cc:366] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.849999 6908 raft_consensus.cc:705] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.849997 6916 tablet_bootstrap.cc:439] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.850090 6913 raft_consensus.cc:340] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.841537 6915 raft_consensus.cc:340] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.850492 6913 raft_consensus.cc:366] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.850551 6915 raft_consensus.cc:366] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.850589 6913 raft_consensus.cc:705] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.850620 6910 ts_tablet_manager.cc:1087] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0720 00:44:16.850649 6915 raft_consensus.cc:705] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.850312 6916 ts_tablet_manager.cc:1059] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0720 00:44:16.850862 6910 tablet_bootstrap.cc:439] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.850862 6915 consensus_queue.cc:229] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.851313 6916 raft_consensus.cc:340] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.851799 6916 raft_consensus.cc:366] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.851886 6916 raft_consensus.cc:705] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.850432 6914 tablet_bootstrap.cc:439] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.852171 6910 tablet_bootstrap.cc:590] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. W0720 00:44:16.852237 6885 tablet.cc:1799] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:16.852661 6912 raft_consensus.cc:366] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.852715 6912 raft_consensus.cc:705] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.853236 6914 tablet_bootstrap.cc:590] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.853571 6912 consensus_queue.cc:229] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.851619 6913 consensus_queue.cc:229] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.854307 6911 tablet_bootstrap.cc:439] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.854480 6913 ts_tablet_manager.cc:1087] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s I0720 00:44:16.854737 6913 tablet_bootstrap.cc:439] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.854735 6915 ts_tablet_manager.cc:1087] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s I0720 00:44:16.854933 6915 tablet_bootstrap.cc:439] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.856010 6911 ts_tablet_manager.cc:1059] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s I0720 00:44:16.856210 6913 tablet_bootstrap.cc:590] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.856416 6915 tablet_bootstrap.cc:590] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.856638 6911 raft_consensus.cc:340] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.856909 6911 raft_consensus.cc:366] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.856990 6911 raft_consensus.cc:705] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.854899 6916 consensus_queue.cc:229] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.855178 6912 ts_tablet_manager.cc:1087] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s I0720 00:44:16.857378 6912 tablet_bootstrap.cc:439] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.857420 6916 ts_tablet_manager.cc:1087] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s I0720 00:44:16.857702 6909 tablet_bootstrap.cc:439] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.857741 6916 tablet_bootstrap.cc:439] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Bootstrap starting. I0720 00:44:16.857869 6909 ts_tablet_manager.cc:1059] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s I0720 00:44:16.851127 6908 consensus_queue.cc:229] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.858639 6912 tablet_bootstrap.cc:590] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.858762 6908 ts_tablet_manager.cc:1087] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s I0720 00:44:16.859001 6916 tablet_bootstrap.cc:590] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No blocks or log segments found. Creating new log. I0720 00:44:16.858803 6911 consensus_queue.cc:229] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.860234 6910 tablet_bootstrap.cc:439] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.860405 6910 ts_tablet_manager.cc:1059] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s I0720 00:44:16.859712 6914 tablet_bootstrap.cc:439] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.860708 6914 ts_tablet_manager.cc:1059] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s I0720 00:44:16.861057 6910 raft_consensus.cc:340] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.861311 6910 raft_consensus.cc:366] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.861384 6910 raft_consensus.cc:705] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.861275 6914 raft_consensus.cc:340] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.861626 6910 consensus_queue.cc:229] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.862155 6914 raft_consensus.cc:366] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.862164 6910 ts_tablet_manager.cc:1087] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:16.862264 6914 raft_consensus.cc:705] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.862372 6910 tablet_bootstrap.cc:439] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.862102 6913 tablet_bootstrap.cc:439] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.862426 6908 tablet_bootstrap.cc:439] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.862534 6913 ts_tablet_manager.cc:1059] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s I0720 00:44:16.863124 6913 raft_consensus.cc:340] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.863689 6910 tablet_bootstrap.cc:590] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.863896 6913 raft_consensus.cc:366] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.864012 6913 raft_consensus.cc:705] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.863517 6914 consensus_queue.cc:229] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.864979 6914 ts_tablet_manager.cc:1087] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0720 00:44:16.865141 6910 tablet_bootstrap.cc:439] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.865289 6910 ts_tablet_manager.cc:1059] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0720 00:44:16.865201 6913 consensus_queue.cc:229] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.865710 6913 ts_tablet_manager.cc:1087] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0720 00:44:16.865952 6910 raft_consensus.cc:340] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.866192 6910 raft_consensus.cc:366] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.866283 6910 raft_consensus.cc:705] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.866495 6910 consensus_queue.cc:229] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.866705 6916 tablet_bootstrap.cc:439] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: No bootstrap required, opened a new log I0720 00:44:16.866849 6916 ts_tablet_manager.cc:1059] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s I0720 00:44:16.867470 6916 raft_consensus.cc:340] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.867660 6911 ts_tablet_manager.cc:1087] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s I0720 00:44:16.867671 6916 raft_consensus.cc:366] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.867733 6916 raft_consensus.cc:705] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Initialized, Role: FOLLOWER I0720 00:44:16.867949 6916 consensus_queue.cc:229] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.868474 6916 ts_tablet_manager.cc:1087] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:16.868695 6908 tablet_bootstrap.cc:590] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.862830 6909 raft_consensus.cc:340] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.868973 6909 raft_consensus.cc:366] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.869019 6909 raft_consensus.cc:705] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.869230 6909 consensus_queue.cc:229] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.869022 6910 ts_tablet_manager.cc:1087] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s I0720 00:44:16.869777 6911 tablet_bootstrap.cc:439] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426: Bootstrap starting. I0720 00:44:16.871553 6909 ts_tablet_manager.cc:1087] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s I0720 00:44:16.872721 6909 tablet_bootstrap.cc:439] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.872843 6911 tablet_bootstrap.cc:590] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426: No blocks or log segments found. Creating new log. I0720 00:44:16.874202 6912 tablet_bootstrap.cc:439] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.874390 6912 ts_tablet_manager.cc:1059] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s I0720 00:44:16.874701 6909 tablet_bootstrap.cc:590] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.875013 6908 tablet_bootstrap.cc:439] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.875063 6912 raft_consensus.cc:340] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.875222 6908 ts_tablet_manager.cc:1059] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s I0720 00:44:16.875300 6912 raft_consensus.cc:366] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.875407 6912 raft_consensus.cc:705] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.875524 6911 tablet_bootstrap.cc:439] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426: No bootstrap required, opened a new log I0720 00:44:16.875756 6911 ts_tablet_manager.cc:1059] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s I0720 00:44:16.875782 6915 tablet_bootstrap.cc:439] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.875787 6912 consensus_queue.cc:229] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.875950 6908 raft_consensus.cc:340] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.876163 6908 raft_consensus.cc:366] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.876241 6908 raft_consensus.cc:705] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.876469 6915 ts_tablet_manager.cc:1059] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.000s I0720 00:44:16.876526 6908 consensus_queue.cc:229] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.876757 6911 raft_consensus.cc:340] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.877032 6911 raft_consensus.cc:366] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.877120 6911 raft_consensus.cc:705] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Initialized, Role: FOLLOWER I0720 00:44:16.877318 6911 consensus_queue.cc:229] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.877496 6908 ts_tablet_manager.cc:1087] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0720 00:44:16.877712 6908 tablet_bootstrap.cc:439] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: Bootstrap starting. I0720 00:44:16.877955 6912 ts_tablet_manager.cc:1087] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I0720 00:44:16.878391 6911 ts_tablet_manager.cc:1087] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0720 00:44:16.878279 6915 raft_consensus.cc:340] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.878527 6915 raft_consensus.cc:366] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.878593 6915 raft_consensus.cc:705] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.878784 6915 consensus_queue.cc:229] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.879045 6908 tablet_bootstrap.cc:590] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: No blocks or log segments found. Creating new log. I0720 00:44:16.879382 6909 tablet_bootstrap.cc:439] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.879500 6909 ts_tablet_manager.cc:1059] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s I0720 00:44:16.880349 6915 ts_tablet_manager.cc:1087] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s I0720 00:44:16.880797 6909 raft_consensus.cc:340] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.881002 6909 raft_consensus.cc:366] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.881064 6909 raft_consensus.cc:705] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.881234 6909 consensus_queue.cc:229] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.881582 6909 ts_tablet_manager.cc:1087] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:16.882586 6908 tablet_bootstrap.cc:439] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: No bootstrap required, opened a new log I0720 00:44:16.882717 6908 ts_tablet_manager.cc:1059] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s I0720 00:44:16.883337 6908 raft_consensus.cc:340] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.883481 6908 raft_consensus.cc:366] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:16.883510 6908 raft_consensus.cc:705] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Initialized, Role: FOLLOWER I0720 00:44:16.883620 6908 consensus_queue.cc:229] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.883908 6908 ts_tablet_manager.cc:1087] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0720 00:44:16.930312 6926 raft_consensus.cc:472] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.930534 6926 raft_consensus.cc:494] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.930646 6929 raft_consensus.cc:472] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.930821 6929 raft_consensus.cc:494] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.931869 6926 leader_election.cc:296] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.931962 6929 leader_election.cc:296] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.935086 6928 raft_consensus.cc:472] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.935288 6928 raft_consensus.cc:494] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.936785 6928 leader_election.cc:296] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519) I0720 00:44:16.939958 6926 raft_consensus.cc:472] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.940090 6926 raft_consensus.cc:494] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.940644 6926 leader_election.cc:296] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.942318 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:16.942787 6717 raft_consensus.cc:2350] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:16.943526 6525 leader_election.cc:310] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.944064 6929 raft_consensus.cc:2644] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.944169 6929 raft_consensus.cc:472] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:16.944237 6929 raft_consensus.cc:2890] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.944941 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:16.945209 6841 raft_consensus.cc:2350] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:16.947451 6929 raft_consensus.cc:494] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.948388 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:16.948554 6717 raft_consensus.cc:2890] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.948459 6592 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27478acfc8ee4027a84efa2c19ee11e9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:16.948732 6592 raft_consensus.cc:2350] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.949142 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "57d8956326d345eba0ced816663169f9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:16.949347 6591 raft_consensus.cc:2350] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.949297 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27478acfc8ee4027a84efa2c19ee11e9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:16.949389 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "57d8956326d345eba0ced816663169f9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:16.949501 6841 raft_consensus.cc:2350] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.949601 6840 raft_consensus.cc:2350] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.950027 6929 leader_election.cc:296] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Requested vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.949950 6648 leader_election.cc:310] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: I0720 00:44:16.950477 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:16.950604 6840 raft_consensus.cc:2890] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.950654 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f49376a672014e499d7fc53f9325c6f6" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:16.950779 6648 leader_election.cc:310] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: I0720 00:44:16.950809 6591 raft_consensus.cc:2350] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 0. I0720 00:44:16.951239 6927 raft_consensus.cc:2644] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.951273 6772 leader_election.cc:310] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.951587 6926 raft_consensus.cc:2644] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.951658 6926 raft_consensus.cc:472] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:16.951730 6928 raft_consensus.cc:2644] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.951750 6926 raft_consensus.cc:2890] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.951814 6928 raft_consensus.cc:472] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:16.951889 6928 raft_consensus.cc:2890] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.952685 6717 raft_consensus.cc:2350] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:16.953001 6840 raft_consensus.cc:2350] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:16.953022 6525 leader_election.cc:310] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.951985 6927 raft_consensus.cc:472] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:16.953435 6929 raft_consensus.cc:2644] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:16.953555 6929 raft_consensus.cc:667] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Becoming Leader. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Running, Role: LEADER I0720 00:44:16.953963 6928 raft_consensus.cc:494] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.953936 6929 consensus_queue.cc:206] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.954536 6928 leader_election.cc:296] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 election: Requested vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519) I0720 00:44:16.955025 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f49376a672014e499d7fc53f9325c6f6" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:16.955157 6591 raft_consensus.cc:2890] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.956353 6927 raft_consensus.cc:2890] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.957398 6591 raft_consensus.cc:2350] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 1. I0720 00:44:16.957741 6772 leader_election.cc:310] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.958019 6928 raft_consensus.cc:2644] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:16.958499 6928 raft_consensus.cc:667] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 LEADER]: Becoming Leader. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Running, Role: LEADER I0720 00:44:16.958463 6926 raft_consensus.cc:494] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.958250 6467 catalog_manager.cc:4067] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 reported cstate change: term changed from 0 to 1, leader changed from to b88011e9e0d24f79882735c95c8fc275 (127.0.58.129). New cstate: current_term: 1 leader_uuid: "b88011e9e0d24f79882735c95c8fc275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:16.959051 6926 leader_election.cc:296] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.959048 6928 consensus_queue.cc:206] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.959662 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "57d8956326d345eba0ced816663169f9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:16.959821 6840 raft_consensus.cc:2890] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.959898 6927 raft_consensus.cc:494] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.960424 6927 leader_election.cc:296] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.961000 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27478acfc8ee4027a84efa2c19ee11e9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:16.961145 6841 raft_consensus.cc:2890] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.961593 6592 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "27478acfc8ee4027a84efa2c19ee11e9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:16.961725 6592 raft_consensus.cc:2890] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.961822 6840 raft_consensus.cc:2350] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:16.962203 6648 leader_election.cc:310] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: I0720 00:44:16.962466 6927 raft_consensus.cc:2644] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:16.962584 6927 raft_consensus.cc:667] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:16.962870 6927 consensus_queue.cc:206] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.963366 6841 raft_consensus.cc:2350] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:16.963754 6592 raft_consensus.cc:2350] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:16.963798 6648 leader_election.cc:310] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: I0720 00:44:16.964385 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "57d8956326d345eba0ced816663169f9" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:16.964499 6931 raft_consensus.cc:472] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.964546 6591 raft_consensus.cc:2890] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.966640 6467 catalog_manager.cc:4067] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: term changed from 0 to 1, leader changed from to 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131). New cstate: current_term: 1 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:16.966802 6591 raft_consensus.cc:2350] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:16.967742 6926 raft_consensus.cc:2644] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:16.964632 6931 raft_consensus.cc:494] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.968621 6931 leader_election.cc:296] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.968842 6926 raft_consensus.cc:667] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:16.969070 6926 consensus_queue.cc:206] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.974680 6716 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f49376a672014e499d7fc53f9325c6f6" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:16.974948 6716 raft_consensus.cc:2890] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.974979 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "f49376a672014e499d7fc53f9325c6f6" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:16.977447 6716 raft_consensus.cc:2350] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 1. I0720 00:44:16.978157 6717 raft_consensus.cc:2325] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa for term 1 because replica is already servicing an update from a current leader or another vote. I0720 00:44:16.978369 6929 raft_consensus.cc:472] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.978499 6929 raft_consensus.cc:494] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.982223 6929 leader_election.cc:296] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.982664 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:16.982851 6840 raft_consensus.cc:2350] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:16.983216 6523 leader_election.cc:310] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.983521 6929 raft_consensus.cc:2644] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.983582 6929 raft_consensus.cc:472] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:16.983652 6929 raft_consensus.cc:2890] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.984706 6930 raft_consensus.cc:472] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.984875 6930 raft_consensus.cc:494] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.985486 6930 leader_election.cc:296] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.985801 6929 raft_consensus.cc:494] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.986510 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:16.986688 6840 raft_consensus.cc:2350] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:16.986874 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:16.986933 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:16.987102 6841 raft_consensus.cc:2890] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.987129 6717 raft_consensus.cc:2350] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:16.987789 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:16.987972 6717 raft_consensus.cc:2890] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.989097 6841 raft_consensus.cc:2350] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:16.989460 6523 leader_election.cc:310] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.989753 6523 leader_election.cc:310] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.989073 6716 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:16.990111 6930 raft_consensus.cc:2644] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.990180 6930 raft_consensus.cc:472] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:16.990240 6930 raft_consensus.cc:2890] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.990229 6716 raft_consensus.cc:2350] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:16.990854 6717 raft_consensus.cc:2350] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:16.992148 6930 raft_consensus.cc:494] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.992637 6930 leader_election.cc:296] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Requested vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.992775 6930 raft_consensus.cc:2644] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:16.992897 6930 raft_consensus.cc:667] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Becoming Leader. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Running, Role: LEADER I0720 00:44:16.993088 6930 consensus_queue.cc:206] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.993660 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:16.993793 6717 raft_consensus.cc:2890] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:16.994787 6467 catalog_manager.cc:4067] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 0 to 1, leader changed from to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:16.995657 6717 raft_consensus.cc:2350] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:16.995986 6525 leader_election.cc:310] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:16.996836 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7139dc2492b147989250601c720f324c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:16.997004 6841 raft_consensus.cc:2350] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.997321 6927 raft_consensus.cc:472] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:16.997424 6927 raft_consensus.cc:494] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.997946 6927 leader_election.cc:296] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.998498 6648 leader_election.cc:310] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: I0720 00:44:16.998649 6929 leader_election.cc:296] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Requested vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:16.999037 6592 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "49ad7f72eaa04a14bc64fa4f6cbce73c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:16.999218 6592 raft_consensus.cc:2350] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.999374 6929 raft_consensus.cc:2644] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:16.999490 6929 raft_consensus.cc:667] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Becoming Leader. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Running, Role: LEADER I0720 00:44:16.999492 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7139dc2492b147989250601c720f324c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:16.999636 6591 raft_consensus.cc:2350] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:16.999657 6929 consensus_queue.cc:206] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:16.999931 6927 raft_consensus.cc:2644] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:16.999997 6927 raft_consensus.cc:472] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:17.000049 6927 raft_consensus.cc:2890] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.000308 6470 catalog_manager.cc:4067] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 0 to 1, leader changed from to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:17.002494 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "49ad7f72eaa04a14bc64fa4f6cbce73c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:17.002662 6841 raft_consensus.cc:2350] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 0. I0720 00:44:17.003011 6648 leader_election.cc:310] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:17.003567 6926 raft_consensus.cc:2644] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:17.003624 6926 raft_consensus.cc:472] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:17.003684 6926 raft_consensus.cc:2890] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.005604 6926 raft_consensus.cc:494] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.006175 6926 leader_election.cc:296] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:17.002019 6927 raft_consensus.cc:494] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.006654 6927 leader_election.cc:296] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:17.006947 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "49ad7f72eaa04a14bc64fa4f6cbce73c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:17.007122 6591 raft_consensus.cc:2890] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.007201 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "49ad7f72eaa04a14bc64fa4f6cbce73c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:17.007354 6841 raft_consensus.cc:2890] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.009485 6591 raft_consensus.cc:2350] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:17.009640 6841 raft_consensus.cc:2350] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:17.010331 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7139dc2492b147989250601c720f324c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:17.010511 6841 raft_consensus.cc:2890] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.011919 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:17.012056 6840 raft_consensus.cc:2890] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.014297 6840 raft_consensus.cc:2350] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:17.014497 6841 raft_consensus.cc:2350] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:17.014981 6648 leader_election.cc:310] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:17.015482 6648 leader_election.cc:310] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: I0720 00:44:17.015550 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7139dc2492b147989250601c720f324c" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:17.015723 6591 raft_consensus.cc:2890] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.015738 6926 raft_consensus.cc:2644] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:17.015841 6926 raft_consensus.cc:667] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:17.016022 6926 consensus_queue.cc:206] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.017422 6926 raft_consensus.cc:2644] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:17.017535 6926 raft_consensus.cc:667] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:17.017706 6926 consensus_queue.cc:206] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.018051 6591 raft_consensus.cc:2350] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. I0720 00:44:17.019747 6467 catalog_manager.cc:4067] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 reported cstate change: term changed from 0 to 1, leader changed from to b88011e9e0d24f79882735c95c8fc275 (127.0.58.129). New cstate: current_term: 1 leader_uuid: "b88011e9e0d24f79882735c95c8fc275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:17.020192 6470 catalog_manager.cc:4067] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 0 to 1, leader changed from to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:17.020591 6470 catalog_manager.cc:4067] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 0 to 1, leader changed from to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:17.023813 6470 catalog_manager.cc:4067] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 reported cstate change: term changed from 0 to 1, leader changed from to b88011e9e0d24f79882735c95c8fc275 (127.0.58.129). New cstate: current_term: 1 leader_uuid: "b88011e9e0d24f79882735c95c8fc275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } W0720 00:44:17.069506 6636 tablet.cc:1799] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:17.157622 6930 raft_consensus.cc:472] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:17.157809 6930 raft_consensus.cc:494] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.158435 6930 leader_election.cc:296] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:17.159068 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:17.159085 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:17.159267 6717 raft_consensus.cc:2350] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:17.159271 6841 raft_consensus.cc:2350] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 0. I0720 00:44:17.159651 6523 leader_election.cc:310] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:17.159941 6930 raft_consensus.cc:2644] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:17.159991 6930 raft_consensus.cc:472] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:17.160035 6930 raft_consensus.cc:2890] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.163602 6930 raft_consensus.cc:494] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.164217 6930 leader_election.cc:296] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Requested vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:17.164664 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:17.164734 6841 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "b88011e9e0d24f79882735c95c8fc275" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:17.164834 6717 raft_consensus.cc:2890] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.164898 6841 raft_consensus.cc:2890] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.166399 6717 raft_consensus.cc:2350] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:17.166414 6841 raft_consensus.cc:2350] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b88011e9e0d24f79882735c95c8fc275 in term 1. I0720 00:44:17.166759 6523 leader_election.cc:310] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:17.167012 6930 raft_consensus.cc:2644] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:17.167134 6930 raft_consensus.cc:667] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Becoming Leader. State: Replica: b88011e9e0d24f79882735c95c8fc275, State: Running, Role: LEADER I0720 00:44:17.167309 6930 consensus_queue.cc:206] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.169483 6470 catalog_manager.cc:4067] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 reported cstate change: term changed from 0 to 1, leader changed from to b88011e9e0d24f79882735c95c8fc275 (127.0.58.129). New cstate: current_term: 1 leader_uuid: "b88011e9e0d24f79882735c95c8fc275" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:17.198493 6928 raft_consensus.cc:472] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:17.198668 6928 raft_consensus.cc:494] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.199270 6928 leader_election.cc:296] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) I0720 00:44:17.199903 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "79f36e06977e420f94ab39786832ecc9" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:17.199901 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "79f36e06977e420f94ab39786832ecc9" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:17.200075 6591 raft_consensus.cc:2350] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 0. I0720 00:44:17.200086 6717 raft_consensus.cc:2350] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 0. I0720 00:44:17.200491 6772 leader_election.cc:310] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:17.200798 6928 raft_consensus.cc:2644] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:17.200860 6928 raft_consensus.cc:472] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:17.200908 6928 raft_consensus.cc:2890] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.202491 6928 raft_consensus.cc:494] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.202941 6928 leader_election.cc:296] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) I0720 00:44:17.203371 6591 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "79f36e06977e420f94ab39786832ecc9" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:17.203402 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "79f36e06977e420f94ab39786832ecc9" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:17.203505 6591 raft_consensus.cc:2890] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.203538 6717 raft_consensus.cc:2890] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:17.205026 6717 raft_consensus.cc:2350] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 1. I0720 00:44:17.205030 6591 raft_consensus.cc:2350] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 1. I0720 00:44:17.205372 6772 leader_election.cc:310] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275; no voters: I0720 00:44:17.205627 6928 raft_consensus.cc:2644] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:17.205755 6928 raft_consensus.cc:667] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 LEADER]: Becoming Leader. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Running, Role: LEADER I0720 00:44:17.205936 6928 consensus_queue.cc:206] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:17.208101 6470 catalog_manager.cc:4067] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: term changed from 0 to 1, leader changed from to 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131). New cstate: current_term: 1 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:17.369014 6930 consensus_queue.cc:984] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0720 00:44:17.372540 6926 consensus_queue.cc:984] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.377614 6716 raft_consensus.cc:1184] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.378593 6928 consensus_queue.cc:984] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.388598 6967 consensus_queue.cc:984] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.390923 6592 raft_consensus.cc:1184] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.391561 6932 consensus_queue.cc:984] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.394917 6840 raft_consensus.cc:1184] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.399682 6841 raft_consensus.cc:1184] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer b88011e9e0d24f79882735c95c8fc275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.400094 6967 consensus_queue.cc:984] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.400550 6716 raft_consensus.cc:1184] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer b88011e9e0d24f79882735c95c8fc275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.400969 6930 consensus_queue.cc:984] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.401324 6839 raft_consensus.cc:1184] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer b88011e9e0d24f79882735c95c8fc275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.402981 6930 consensus_queue.cc:984] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0720 00:44:17.405889 6716 raft_consensus.cc:1184] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer b88011e9e0d24f79882735c95c8fc275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.407305 6964 consensus_queue.cc:984] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.410717 6930 consensus_queue.cc:984] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s I0720 00:44:17.425976 7000 mvcc.cc:203] Tried to move safe_time back from 6404437841455169536 to 6404437839722205184. Current Snapshot: MvccSnapshot[committed={T|T < 6404437841455169536}] I0720 00:44:17.426571 6841 raft_consensus.cc:1184] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.426625 6840 raft_consensus.cc:1184] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.428546 6927 consensus_queue.cc:984] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:44:17.429517 6590 raft_consensus.cc:1184] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.429541 6926 consensus_queue.cc:984] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.430033 6589 raft_consensus.cc:1184] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.430624 6589 raft_consensus.cc:1184] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.431254 6926 consensus_queue.cc:984] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.431429 6964 consensus_queue.cc:984] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.440522 6964 consensus_queue.cc:984] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.441038 6839 raft_consensus.cc:1184] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.441525 6964 consensus_queue.cc:984] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.452445 6587 raft_consensus.cc:1184] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.453088 7019 consensus_queue.cc:984] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.453155 6987 mvcc.cc:203] Tried to move safe_time back from 6404437841455169536 to 6404437839722205184. Current Snapshot: MvccSnapshot[committed={T|T < 6404437841455169536 or (T in {6404437841455169536})}] I0720 00:44:17.453459 7018 consensus_queue.cc:984] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.489675 6715 raft_consensus.cc:1184] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer b88011e9e0d24f79882735c95c8fc275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.492393 7019 consensus_queue.cc:984] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0720 00:44:17.500878 7025 mvcc.cc:203] Tried to move safe_time back from 6404437841632251904 to 6404437839946485760. Current Snapshot: MvccSnapshot[committed={T|T < 6404437841409818624}] I0720 00:44:17.526476 6716 raft_consensus.cc:1184] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:17.529786 6928 consensus_queue.cc:984] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:17.838778 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000282 I0720 00:44:17.855329 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000284 I0720 00:44:17.950106 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.080s user 0.023s sys 0.005s Metrics: {"bytes_written":6699,"cfile_init":1,"compiler_manager_pool.queue_time_us":5190,"data dirs.queue_time_us":6473,"data dirs.run_cpu_time_us":377,"data dirs.run_wall_time_us":8709,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"rows_written":79,"thread_start_us":5908,"threads_started":2} I0720 00:44:17.955204 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000311 W0720 00:44:17.968538 6774 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:42329, user_credentials={real_user=slave}} blocked reactor thread for 68325.5us I0720 00:44:18.072459 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.233s user 0.029s sys 0.001s Metrics: {"bytes_written":6458,"cfile_init":1,"compiler_manager_pool.queue_time_us":718,"data dirs.queue_time_us":3101,"data dirs.run_cpu_time_us":140,"data dirs.run_wall_time_us":4594,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"rows_written":70,"thread_start_us":5766,"threads_started":2} I0720 00:44:18.077298 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000346 I0720 00:44:18.092259 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000349 I0720 00:44:18.134459 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.042s user 0.014s sys 0.002s Metrics: {"bytes_written":7676,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":515,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":759,"lbm_writes_lt_1ms":23,"rows_written":115} I0720 00:44:18.139001 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000360 I0720 00:44:18.160542 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.205s user 0.008s sys 0.004s Metrics: {"bytes_written":6882,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":124,"data dirs.run_wall_time_us":463,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"rows_written":85} I0720 00:44:18.161698 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000367 W0720 00:44:18.170374 6894 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.58.131:38005, user_credentials={real_user=slave}} blocked reactor thread for 52061.6us I0720 00:44:18.182655 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.043s user 0.013s sys 0.004s Metrics: {"bytes_written":7448,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":477,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"rows_written":106} I0720 00:44:18.183876 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000370 I0720 00:44:18.199301 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":7510,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":115,"data dirs.run_wall_time_us":745,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"rows_written":110} I0720 00:44:18.200438 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000374 I0720 00:44:18.278064 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.116s user 0.006s sys 0.009s Metrics: {"bytes_written":6646,"cfile_init":1,"data dirs.queue_time_us":638,"data dirs.run_cpu_time_us":153,"data dirs.run_wall_time_us":1492,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"rows_written":76,"thread_start_us":301,"threads_started":1} I0720 00:44:18.279289 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 935 bytes on disk I0720 00:44:18.280030 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:18.280752 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 861 bytes on disk I0720 00:44:18.281270 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:18.281963 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000399 I0720 00:44:18.297294 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.015s user 0.006s sys 0.006s Metrics: {"bytes_written":7333,"cfile_init":1,"data dirs.queue_time_us":38,"data dirs.run_cpu_time_us":124,"data dirs.run_wall_time_us":519,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":642,"lbm_writes_lt_1ms":23,"rows_written":104} I0720 00:44:18.298907 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 1118 bytes on disk I0720 00:44:18.299813 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:18.300572 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000403 I0720 00:44:18.316165 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.238s user 0.035s sys 0.014s Metrics: {"bytes_written":7495,"cfile_init":1,"compiler_manager_pool.queue_time_us":48986,"data dirs.queue_time_us":8472,"data dirs.run_cpu_time_us":189,"data dirs.run_wall_time_us":729,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":666,"lbm_writes_lt_1ms":23,"rows_written":109,"thread_start_us":53426,"threads_started":2} I0720 00:44:18.318928 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.017s user 0.012s sys 0.003s Metrics: {"bytes_written":7852,"cfile_init":1,"data dirs.queue_time_us":403,"data dirs.run_cpu_time_us":126,"data dirs.run_wall_time_us":2701,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":766,"lbm_writes_lt_1ms":23,"rows_written":122,"thread_start_us":479,"threads_started":1} I0720 00:44:18.319955 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 1292 bytes on disk I0720 00:44:18.320523 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:18.321421 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000407 I0720 00:44:18.340780 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.019s user 0.016s sys 0.000s Metrics: {"bytes_written":8060,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":487,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"rows_written":131} I0720 00:44:18.341945 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 1388 bytes on disk I0720 00:44:18.342540 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4} I0720 00:44:18.343286 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 860 bytes on disk I0720 00:44:18.343806 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:18.344521 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000413 I0720 00:44:18.373088 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.028s user 0.010s sys 0.004s Metrics: {"bytes_written":7946,"cfile_init":1,"data dirs.queue_time_us":1427,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":522,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":23,"rows_written":127} I0720 00:44:18.379014 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 1321 bytes on disk I0720 00:44:18.379633 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:18.380419 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000421 I0720 00:44:18.392787 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000432 I0720 00:44:18.405282 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.203s user 0.011s sys 0.000s Metrics: {"bytes_written":7732,"cfile_init":1,"data dirs.queue_time_us":4360,"data dirs.run_cpu_time_us":131,"data dirs.run_wall_time_us":2432,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":504,"lbm_writes_lt_1ms":23,"rows_written":120,"thread_start_us":24672,"threads_started":1} I0720 00:44:18.422186 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.042s user 0.013s sys 0.003s Metrics: {"bytes_written":7872,"cfile_init":1,"data dirs.queue_time_us":2731,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":567,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"rows_written":124} I0720 00:44:18.423354 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000433 I0720 00:44:18.431241 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 1278 bytes on disk I0720 00:44:18.432464 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:18.443094 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000439 I0720 00:44:18.479311 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.056s user 0.013s sys 0.002s Metrics: {"bytes_written":7878,"cfile_init":1,"data dirs.queue_time_us":4364,"data dirs.run_cpu_time_us":131,"data dirs.run_wall_time_us":644,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":720,"lbm_writes_lt_1ms":23,"rows_written":124} I0720 00:44:18.480515 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000448 I0720 00:44:18.504122 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.023s user 0.012s sys 0.003s Metrics: {"bytes_written":8369,"cfile_init":1,"data dirs.queue_time_us":38,"data dirs.run_cpu_time_us":94,"data dirs.run_wall_time_us":390,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":758,"lbm_writes_lt_1ms":23,"rows_written":144} I0720 00:44:18.505295 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 1516 bytes on disk I0720 00:44:18.505851 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:18.506690 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 1296 bytes on disk I0720 00:44:18.507200 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:18.507936 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 1290 bytes on disk I0720 00:44:18.508419 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:18.544991 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.152s user 0.014s sys 0.001s Metrics: {"bytes_written":7528,"cfile_init":1,"data dirs.queue_time_us":6669,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":5809,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":759,"lbm_writes_lt_1ms":23,"rows_written":112,"thread_start_us":10331,"threads_started":1} I0720 00:44:18.550947 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000474 I0720 00:44:18.612992 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.062s user 0.016s sys 0.000s Metrics: {"bytes_written":9281,"cfile_init":1,"data dirs.queue_time_us":62,"data dirs.run_cpu_time_us":136,"data dirs.run_wall_time_us":514,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":629,"lbm_writes_lt_1ms":23,"rows_written":177} I0720 00:44:18.618968 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000490 I0720 00:44:18.621333 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.178s user 0.016s sys 0.000s Metrics: {"bytes_written":7878,"cfile_init":1,"data dirs.queue_time_us":15511,"data dirs.run_cpu_time_us":122,"data dirs.run_wall_time_us":765,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6887,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":124} I0720 00:44:18.642259 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 1296 bytes on disk I0720 00:44:18.642822 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:18.650676 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000495 I0720 00:44:18.726688 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.076s user 0.017s sys 0.000s Metrics: {"bytes_written":8698,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":150,"data dirs.run_wall_time_us":543,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":738,"lbm_writes_lt_1ms":23,"rows_written":155} I0720 00:44:18.727885 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000516 I0720 00:44:18.749687 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.022s user 0.019s sys 0.000s Metrics: {"bytes_written":9721,"cfile_init":1,"data dirs.queue_time_us":747,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":3180,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":718,"lbm_writes_lt_1ms":23,"rows_written":195,"thread_start_us":1166,"threads_started":1} I0720 00:44:18.750867 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 1983 bytes on disk I0720 00:44:18.751446 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:18.752228 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000521 I0720 00:44:18.869539 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.250s user 0.015s sys 0.003s Metrics: {"bytes_written":9288,"cfile_init":1,"data dirs.queue_time_us":21329,"data dirs.run_cpu_time_us":175,"data dirs.run_wall_time_us":9482,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"rows_written":178,"thread_start_us":77374,"threads_started":1} I0720 00:44:18.871336 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 1844 bytes on disk I0720 00:44:18.872220 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4} I0720 00:44:18.874761 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000560 I0720 00:44:18.907582 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.155s user 0.016s sys 0.000s Metrics: {"bytes_written":8897,"cfile_init":1,"data dirs.queue_time_us":49,"data dirs.run_cpu_time_us":172,"data dirs.run_wall_time_us":864,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":32453,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":163} I0720 00:44:18.908780 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000564 I0720 00:44:19.010622 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000298 I0720 00:44:19.059274 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.048s user 0.025s sys 0.016s Metrics: {"bytes_written":8732,"cfile_init":1,"data dirs.queue_time_us":25489,"data dirs.run_cpu_time_us":170,"data dirs.run_wall_time_us":1783,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1945,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":157,"thread_start_us":25347,"threads_started":1} W0720 00:44:19.060453 6648 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:41519, user_credentials={real_user=slave}} blocked reactor thread for 70179.5us I0720 00:44:19.060895 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 1631 bytes on disk I0720 00:44:19.061496 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:19.062270 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.989267 I0720 00:44:19.033314 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.158s user 0.017s sys 0.002s Metrics: {"bytes_written":9540,"cfile_init":1,"data dirs.queue_time_us":73,"data dirs.run_cpu_time_us":133,"data dirs.run_wall_time_us":532,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":786,"lbm_writes_lt_1ms":23,"rows_written":189} I0720 00:44:19.111644 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 1915 bytes on disk I0720 00:44:19.111721 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.049s user 0.044s sys 0.004s Metrics: {"bytes_written":10764,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4522,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13292,"cfile_init":5,"data dirs.queue_time_us":25398,"data dirs.run_cpu_time_us":417,"data dirs.run_wall_time_us":6351,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":870,"lbm_reads_lt_1ms":46,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":236,"spinlock_wait_cycles":7040,"thread_start_us":25251,"threads_started":2} I0720 00:44:19.112257 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:19.112912 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 2372 bytes on disk I0720 00:44:19.113454 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:19.119035 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 1818 bytes on disk I0720 00:44:19.123383 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:19.125037 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000626 I0720 00:44:19.172840 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.264s user 0.021s sys 0.000s Metrics: {"bytes_written":10163,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":570,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"rows_written":212} I0720 00:44:19.173946 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000306 I0720 00:44:19.200603 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.026s user 0.014s sys 0.000s Metrics: {"bytes_written":8721,"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":150,"data dirs.run_wall_time_us":528,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1015,"lbm_writes_lt_1ms":23,"rows_written":157} I0720 00:44:19.202112 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.992075 I0720 00:44:19.222914 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.021s user 0.015s sys 0.004s Metrics: {"bytes_written":10539,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4359,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12731,"cfile_init":7,"data dirs.queue_time_us":942,"data dirs.run_cpu_time_us":584,"data dirs.run_wall_time_us":867,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":789,"lbm_reads_lt_1ms":54,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":227,"thread_start_us":446,"threads_started":1} I0720 00:44:19.224798 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 2279 bytes on disk I0720 00:44:19.235697 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:19.236711 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000306 I0720 00:44:19.255327 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"bytes_written":8139,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":118,"data dirs.run_wall_time_us":512,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":758,"lbm_writes_lt_1ms":23,"rows_written":134} I0720 00:44:19.256594 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.008029 I0720 00:44:19.289204 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.164s user 0.014s sys 0.004s Metrics: {"bytes_written":9932,"cfile_init":1,"data dirs.queue_time_us":19965,"data dirs.run_cpu_time_us":199,"data dirs.run_wall_time_us":9864,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"rows_written":203,"thread_start_us":67157,"threads_started":1} I0720 00:44:19.321979 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.065s user 0.018s sys 0.004s Metrics: {"bytes_written":11133,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4758,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14096,"cfile_init":7,"data dirs.queue_time_us":3549,"data dirs.run_cpu_time_us":620,"data dirs.run_wall_time_us":1042,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1080,"lbm_reads_lt_1ms":54,"lbm_write_time_us":800,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":249,"spinlock_wait_cycles":3456} I0720 00:44:19.323285 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000317 I0720 00:44:19.350533 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.017s user 0.012s sys 0.004s Metrics: {"bytes_written":7932,"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":502,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":795,"lbm_writes_lt_1ms":23,"rows_written":128} I0720 00:44:19.351819 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=1.011018 I0720 00:44:19.364998 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000334 I0720 00:44:19.366561 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 2051 bytes on disk I0720 00:44:19.378535 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:19.379343 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 1215 bytes on disk I0720 00:44:19.379906 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:19.380666 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000695 I0720 00:44:19.391782 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.026s user 0.016s sys 0.001s Metrics: {"bytes_written":9502,"cfile_init":1,"compiler_manager_pool.queue_time_us":25537,"compiler_manager_pool.run_cpu_time_us":50,"compiler_manager_pool.run_wall_time_us":51,"data dirs.queue_time_us":5539,"data dirs.run_cpu_time_us":141,"data dirs.run_wall_time_us":585,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":643,"lbm_writes_lt_1ms":23,"rows_written":186} I0720 00:44:19.393115 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 1901 bytes on disk I0720 00:44:19.393725 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:19.394531 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.014529 I0720 00:44:19.407495 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.027s user 0.025s sys 0.000s Metrics: {"bytes_written":11675,"cfile_init":1,"data dirs.queue_time_us":6778,"data dirs.run_cpu_time_us":149,"data dirs.run_wall_time_us":3445,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"rows_written":271,"thread_start_us":2225,"threads_started":1} I0720 00:44:19.408658 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000702 I0720 00:44:19.423524 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.029s user 0.020s sys 0.002s Metrics: {"bytes_written":11675,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5162,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15248,"cfile_init":5,"data dirs.queue_time_us":447,"data dirs.run_cpu_time_us":600,"data dirs.run_wall_time_us":1019,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":770,"lbm_reads_lt_1ms":46,"lbm_write_time_us":589,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":271} I0720 00:44:19.424803 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 2709 bytes on disk I0720 00:44:19.425367 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:19.426187 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000319 I0720 00:44:19.444023 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.018s user 0.010s sys 0.004s Metrics: {"bytes_written":8253,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":109,"data dirs.run_wall_time_us":516,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701,"lbm_writes_lt_1ms":23,"rows_written":137} I0720 00:44:19.445291 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=1.013784 I0720 00:44:19.466984 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.021s user 0.012s sys 0.008s Metrics: {"bytes_written":10185,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4101,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12291,"cfile_init":6,"data dirs.queue_time_us":366,"data dirs.run_cpu_time_us":571,"data dirs.run_wall_time_us":979,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":939,"lbm_reads_lt_1ms":50,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":213} I0720 00:44:19.468451 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 2189 bytes on disk I0720 00:44:19.469095 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:19.470002 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000326 I0720 00:44:19.488873 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.019s user 0.013s sys 0.002s Metrics: {"bytes_written":8324,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":134,"data dirs.run_wall_time_us":568,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"rows_written":141} I0720 00:44:19.490202 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 1463 bytes on disk I0720 00:44:19.490815 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:19.491631 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=1.000318 I0720 00:44:19.513761 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.022s user 0.021s sys 0.000s Metrics: {"bytes_written":11000,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4685,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13761,"cfile_init":5,"data dirs.queue_time_us":304,"data dirs.run_cpu_time_us":514,"data dirs.run_wall_time_us":1221,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":869,"lbm_reads_lt_1ms":46,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":245} I0720 00:44:19.515105 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 2476 bytes on disk I0720 00:44:19.515731 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:19.516568 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000333 I0720 00:44:19.523974 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.172s user 0.027s sys 0.011s Metrics: {"bytes_written":10733,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4483,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13227,"cfile_init":7,"data dirs.queue_time_us":347,"data dirs.run_cpu_time_us":485,"data dirs.run_wall_time_us":1311,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1016,"lbm_reads_lt_1ms":54,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":234} I0720 00:44:19.525210 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 2514 bytes on disk I0720 00:44:19.525828 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:19.526674 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000369 I0720 00:44:19.530664 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.014s user 0.009s sys 0.004s Metrics: {"bytes_written":7491,"cfile_init":1,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":807,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"rows_written":109} I0720 00:44:19.531946 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(57d8956326d345eba0ced816663169f9): perf score=1.009920 I0720 00:44:19.544132 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":7920,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":524,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1126,"lbm_writes_lt_1ms":23,"rows_written":126} I0720 00:44:19.545469 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 2499 bytes on disk I0720 00:44:19.546309 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":180,"lbm_reads_lt_1ms":8} I0720 00:44:19.547153 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 2376 bytes on disk I0720 00:44:19.547667 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:19.548409 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(57d8956326d345eba0ced816663169f9): perf score=1.010466 I0720 00:44:19.553723 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.022s user 0.014s sys 0.007s Metrics: {"bytes_written":10634,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4432,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13148,"cfile_init":6,"data dirs.queue_time_us":346,"data dirs.run_cpu_time_us":572,"data dirs.run_wall_time_us":1322,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":901,"lbm_reads_lt_1ms":50,"lbm_write_time_us":640,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":231} I0720 00:44:19.555059 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 2348 bytes on disk I0720 00:44:19.555660 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:19.556529 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000338 I0720 00:44:19.572667 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.016s user 0.013s sys 0.000s Metrics: {"bytes_written":7912,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":595,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"rows_written":126} I0720 00:44:19.573968 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=1.008960 I0720 00:44:19.600354 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.191s user 0.013s sys 0.008s Metrics: {"bytes_written":10937,"cfile_init":1,"data dirs.queue_time_us":65,"data dirs.run_cpu_time_us":141,"data dirs.run_wall_time_us":514,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"rows_written":241} I0720 00:44:19.640707 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000766 I0720 00:44:19.705986 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.065s user 0.026s sys 0.012s Metrics: {"bytes_written":11163,"cfile_init":1,"data dirs.queue_time_us":15426,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":2465,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":898,"lbm_writes_lt_1ms":23,"rows_written":250,"thread_start_us":15298,"threads_started":1} I0720 00:44:19.707533 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000784 I0720 00:44:19.715852 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.142s user 0.023s sys 0.000s Metrics: {"bytes_written":11337,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4904,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14460,"cfile_init":6,"data dirs.queue_time_us":317,"data dirs.run_cpu_time_us":527,"data dirs.run_wall_time_us":937,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":2733,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":257} I0720 00:44:19.718924 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000374 I0720 00:44:19.739940 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.032s user 0.021s sys 0.006s Metrics: {"bytes_written":11651,"cfile_init":1,"data dirs.queue_time_us":4211,"data dirs.run_cpu_time_us":152,"data dirs.run_wall_time_us":3479,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":749,"lbm_writes_lt_1ms":23,"rows_written":270,"thread_start_us":3580,"threads_started":1} I0720 00:44:19.741210 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 2710 bytes on disk I0720 00:44:19.741770 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:19.742600 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000396 I0720 00:44:19.763751 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.021s user 0.009s sys 0.011s Metrics: {"bytes_written":10220,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":452,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"rows_written":215} I0720 00:44:19.765439 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 3330 bytes on disk I0720 00:44:19.767477 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.045s user 0.009s sys 0.007s Metrics: {"bytes_written":9493,"cfile_init":1,"data dirs.queue_time_us":3459,"data dirs.run_cpu_time_us":127,"data dirs.run_wall_time_us":556,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5755,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":185} I0720 00:44:19.768738 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 2606 bytes on disk I0720 00:44:19.769654 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:19.770445 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 1893 bytes on disk I0720 00:44:19.770968 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:19.771736 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2): perf score=1.008321 I0720 00:44:19.795542 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.024s user 0.022s sys 0.000s Metrics: {"bytes_written":12693,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5909,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17367,"cfile_init":5,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":490,"data dirs.run_wall_time_us":1032,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":915,"lbm_reads_lt_1ms":46,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":312} I0720 00:44:19.796913 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 3094 bytes on disk I0720 00:44:19.797497 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:19.798331 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000382 I0720 00:44:19.800030 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":157,"lbm_reads_lt_1ms":8} I0720 00:44:19.804365 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2): perf score=1.008182 I0720 00:44:19.815101 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":8721,"cfile_init":1,"data dirs.queue_time_us":74,"data dirs.run_cpu_time_us":117,"data dirs.run_wall_time_us":446,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"rows_written":157} I0720 00:44:19.816354 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.994828 I0720 00:44:19.827533 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.022s user 0.021s sys 0.000s Metrics: {"bytes_written":13039,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6128,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17930,"cfile_init":5,"data dirs.queue_time_us":430,"data dirs.run_cpu_time_us":532,"data dirs.run_wall_time_us":893,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":631,"lbm_reads_lt_1ms":46,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":324} I0720 00:44:19.828871 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000356 I0720 00:44:19.842645 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.026s user 0.021s sys 0.000s Metrics: {"bytes_written":11964,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5342,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15604,"cfile_init":6,"data dirs.queue_time_us":349,"data dirs.run_cpu_time_us":503,"data dirs.run_wall_time_us":836,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":840,"lbm_reads_lt_1ms":50,"lbm_write_time_us":695,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":281} I0720 00:44:19.843966 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000379 I0720 00:44:19.893199 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.064s user 0.016s sys 0.000s Metrics: {"bytes_written":8514,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":101,"data dirs.run_wall_time_us":449,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"rows_written":148} I0720 00:44:19.892912 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.049s user 0.013s sys 0.000s Metrics: {"bytes_written":8556,"cfile_init":1,"data dirs.queue_time_us":37,"data dirs.run_cpu_time_us":101,"data dirs.run_wall_time_us":483,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":742,"lbm_writes_lt_1ms":23,"rows_written":151} I0720 00:44:19.894863 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 2783 bytes on disk I0720 00:44:19.895058 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 3210 bytes on disk I0720 00:44:19.895453 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:19.895634 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:19.896239 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 1549 bytes on disk I0720 00:44:19.896450 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=1.013804 I0720 00:44:19.896737 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:19.897469 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.006140 I0720 00:44:19.920495 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.023s user 0.022s sys 0.000s Metrics: {"bytes_written":11391,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4955,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14599,"cfile_init":6,"data dirs.queue_time_us":299,"data dirs.run_cpu_time_us":496,"data dirs.run_wall_time_us":821,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1000,"lbm_reads_lt_1ms":50,"lbm_write_time_us":723,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":260} I0720 00:44:19.921737 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000363 I0720 00:44:19.921730 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.024s user 0.022s sys 0.000s Metrics: {"bytes_written":11787,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5230,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15298,"cfile_init":5,"data dirs.queue_time_us":295,"data dirs.run_cpu_time_us":489,"data dirs.run_wall_time_us":788,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1032,"lbm_reads_lt_1ms":46,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":275} I0720 00:44:19.922931 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 2729 bytes on disk I0720 00:44:19.924243 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:19.924983 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000395 I0720 00:44:19.943452 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":8327,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":496,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":766,"lbm_writes_lt_1ms":23,"rows_written":141} I0720 00:44:19.944602 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=1.010701 I0720 00:44:19.974422 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.030s user 0.019s sys 0.000s Metrics: {"bytes_written":12029,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5415,"cfile_cache_miss":26,"cfile_cache_miss_bytes":15927,"cfile_init":6,"data dirs.queue_time_us":303,"data dirs.run_cpu_time_us":453,"data dirs.run_wall_time_us":1016,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":877,"lbm_reads_lt_1ms":50,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":285} I0720 00:44:19.975615 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 2862 bytes on disk I0720 00:44:19.976208 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:19.987745 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.436s user 0.017s sys 0.006s Metrics: {"bytes_written":10769,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4522,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13316,"cfile_init":5,"data dirs.queue_time_us":19667,"data dirs.run_cpu_time_us":589,"data dirs.run_wall_time_us":947,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":860,"lbm_reads_lt_1ms":46,"lbm_write_time_us":798,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":236} I0720 00:44:20.002171 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000442 I0720 00:44:20.006130 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.059s user 0.013s sys 0.004s Metrics: {"bytes_written":8463,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":512,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":765,"lbm_writes_lt_1ms":23,"rows_written":146} I0720 00:44:20.007375 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=1.003620 I0720 00:44:20.188062 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.177s user 0.020s sys 0.005s Metrics: {"bytes_written":13023,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6111,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18019,"cfile_init":6,"data dirs.queue_time_us":327,"data dirs.run_cpu_time_us":610,"data dirs.run_wall_time_us":1023,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1093,"lbm_reads_lt_1ms":50,"lbm_write_time_us":749,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":323} I0720 00:44:20.189369 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000367 I0720 00:44:20.208210 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.206s user 0.020s sys 0.000s Metrics: {"bytes_written":9945,"cfile_init":1,"data dirs.queue_time_us":36802,"data dirs.run_cpu_time_us":162,"data dirs.run_wall_time_us":817,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"rows_written":204} I0720 00:44:20.209874 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.020s user 0.013s sys 0.003s Metrics: {"bytes_written":9150,"cfile_init":1,"data dirs.queue_time_us":39,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":500,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":707,"lbm_writes_lt_1ms":23,"rows_written":174} I0720 00:44:20.211153 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 3211 bytes on disk I0720 00:44:20.211699 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:20.212488 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=1.010406 I0720 00:44:20.215932 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=1.008595 I0720 00:44:20.227733 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000324 I0720 00:44:20.279150 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.066s user 0.025s sys 0.000s Metrics: {"bytes_written":13680,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6639,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19463,"cfile_init":6,"data dirs.queue_time_us":323,"data dirs.run_cpu_time_us":587,"data dirs.run_wall_time_us":1017,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":944,"lbm_reads_lt_1ms":50,"lbm_write_time_us":906,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":352} I0720 00:44:20.283097 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 3458 bytes on disk I0720 00:44:20.286550 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:20.287807 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000348 I0720 00:44:20.312688 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.085s user 0.011s sys 0.003s Metrics: {"bytes_written":7942,"cfile_init":1,"data dirs.queue_time_us":13682,"data dirs.run_cpu_time_us":133,"data dirs.run_wall_time_us":701,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"rows_written":127} I0720 00:44:20.314534 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 1325 bytes on disk I0720 00:44:20.315135 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4} I0720 00:44:20.315945 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=1.013241 I0720 00:44:20.328871 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.113s user 0.026s sys 0.000s Metrics: {"bytes_written":13017,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6123,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17905,"cfile_init":6,"data dirs.queue_time_us":8748,"data dirs.run_cpu_time_us":622,"data dirs.run_wall_time_us":1108,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1084,"lbm_reads_lt_1ms":50,"lbm_write_time_us":7896,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":324} I0720 00:44:20.330371 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000475 I0720 00:44:20.367360 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.051s user 0.020s sys 0.004s Metrics: {"bytes_written":14036,"cfile_cache_hit":12,"cfile_cache_hit_bytes":8558,"cfile_cache_miss":24,"cfile_cache_miss_bytes":18216,"cfile_init":5,"data dirs.queue_time_us":405,"data dirs.run_cpu_time_us":483,"data dirs.run_wall_time_us":1201,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1222,"lbm_reads_lt_1ms":44,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"mutex_wait_us":15,"num_input_rowsets":2,"rows_written":363,"spinlock_wait_cycles":3456} I0720 00:44:20.368628 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 3581 bytes on disk I0720 00:44:20.370422 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:20.523794 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.234s user 0.018s sys 0.000s Metrics: {"bytes_written":8309,"cfile_init":1,"data dirs.queue_time_us":36301,"data dirs.run_cpu_time_us":131,"data dirs.run_wall_time_us":593,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7278,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":141} I0720 00:44:20.530714 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 2709 bytes on disk I0720 00:44:20.531953 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:20.532917 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 2632 bytes on disk I0720 00:44:20.538326 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4} I0720 00:44:20.551365 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 2535 bytes on disk I0720 00:44:20.551921 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:20.564810 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=1.011962 I0720 00:44:20.592252 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.262s user 0.008s sys 0.013s Metrics: {"bytes_written":10801,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":135,"data dirs.run_wall_time_us":616,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":713,"lbm_writes_lt_1ms":23,"rows_written":238} I0720 00:44:20.607012 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.000706 I0720 00:44:20.622028 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000342 I0720 00:44:20.643283 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.077s user 0.020s sys 0.003s Metrics: {"bytes_written":13207,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6240,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18202,"cfile_init":6,"data dirs.queue_time_us":311,"data dirs.run_cpu_time_us":538,"data dirs.run_wall_time_us":939,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1007,"lbm_reads_lt_1ms":50,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":330,"spinlock_wait_cycles":4224} I0720 00:44:20.644707 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000376 I0720 00:44:20.648924 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.025s user 0.010s sys 0.004s Metrics: {"bytes_written":8371,"cfile_init":1,"data dirs.queue_time_us":1526,"data dirs.run_cpu_time_us":138,"data dirs.run_wall_time_us":556,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"rows_written":144} I0720 00:44:20.650388 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 1497 bytes on disk I0720 00:44:20.650975 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:20.654738 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.009872 I0720 00:44:20.667639 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.023s user 0.015s sys 0.004s Metrics: {"bytes_written":9508,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":447,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":739,"lbm_writes_lt_1ms":23,"rows_written":187} I0720 00:44:20.669553 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 3256 bytes on disk I0720 00:44:20.670233 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4} I0720 00:44:20.671567 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.005633 I0720 00:44:20.688239 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.033s user 0.023s sys 0.001s Metrics: {"bytes_written":15345,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7786,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22676,"cfile_init":5,"data dirs.queue_time_us":315,"data dirs.run_cpu_time_us":470,"data dirs.run_wall_time_us":910,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1570,"lbm_reads_lt_1ms":46,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":415} I0720 00:44:20.689476 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 4042 bytes on disk I0720 00:44:20.690062 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:20.690877 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000346 I0720 00:44:20.708611 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.037s user 0.021s sys 0.008s Metrics: {"bytes_written":14721,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7330,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21438,"cfile_init":6,"data dirs.queue_time_us":338,"data dirs.run_cpu_time_us":517,"data dirs.run_wall_time_us":914,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":974,"lbm_reads_lt_1ms":50,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":390} I0720 00:44:20.709113 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.018s user 0.010s sys 0.004s Metrics: {"bytes_written":9589,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":507,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"rows_written":192} I0720 00:44:20.709998 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=1.013945 I0720 00:44:20.710127 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000362 I0720 00:44:20.747823 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.038s user 0.022s sys 0.004s Metrics: {"bytes_written":15118,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7605,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22167,"cfile_init":6,"data dirs.queue_time_us":413,"data dirs.run_cpu_time_us":557,"data dirs.run_wall_time_us":989,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":998,"lbm_reads_lt_1ms":50,"lbm_write_time_us":775,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":405} I0720 00:44:20.749122 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000350 I0720 00:44:20.763533 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.053s user 0.016s sys 0.000s Metrics: {"bytes_written":8371,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":500,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":681,"lbm_writes_lt_1ms":23,"rows_written":144} I0720 00:44:20.765113 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.009872 I0720 00:44:20.781308 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.032s user 0.014s sys 0.000s Metrics: {"bytes_written":8709,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":501,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":703,"lbm_writes_lt_1ms":23,"rows_written":156} I0720 00:44:20.782706 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 3956 bytes on disk I0720 00:44:20.783360 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:20.784137 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 1609 bytes on disk I0720 00:44:20.784665 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4} I0720 00:44:20.785369 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=1.005251 I0720 00:44:20.820441 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.034s user 0.018s sys 0.007s Metrics: {"bytes_written":15049,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7532,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22014,"cfile_init":5,"data dirs.queue_time_us":313,"data dirs.run_cpu_time_us":524,"data dirs.run_wall_time_us":982,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":942,"lbm_reads_lt_1ms":46,"lbm_write_time_us":821,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":401} I0720 00:44:20.821789 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000359 I0720 00:44:20.846032 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.024s user 0.011s sys 0.003s Metrics: {"bytes_written":8947,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":117,"data dirs.run_wall_time_us":722,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":819,"lbm_writes_lt_1ms":23,"rows_written":165} I0720 00:44:20.847389 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 3929 bytes on disk I0720 00:44:20.848027 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:20.848803 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(57d8956326d345eba0ced816663169f9): perf score=1.010182 I0720 00:44:20.890569 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.042s user 0.023s sys 0.004s Metrics: {"bytes_written":14893,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7442,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21768,"cfile_init":6,"data dirs.queue_time_us":345,"data dirs.run_cpu_time_us":571,"data dirs.run_wall_time_us":1107,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":989,"lbm_reads_lt_1ms":50,"lbm_write_time_us":858,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":396} I0720 00:44:20.891757 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 3891 bytes on disk I0720 00:44:20.892235 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:20.892974 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000367 I0720 00:44:20.951144 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.058s user 0.016s sys 0.000s Metrics: {"bytes_written":8759,"cfile_init":1,"data dirs.queue_time_us":65,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":444,"drs_written":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"rows_written":160} I0720 00:44:20.952440 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.998607 I0720 00:44:21.030247 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.261s user 0.018s sys 0.007s Metrics: {"bytes_written":15345,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7786,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22676,"cfile_init":6,"data dirs.queue_time_us":359,"data dirs.run_cpu_time_us":568,"data dirs.run_wall_time_us":962,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1011,"lbm_reads_lt_1ms":50,"lbm_write_time_us":706,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":415} I0720 00:44:21.031738 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000397 I0720 00:44:21.040932 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.088s user 0.026s sys 0.000s Metrics: {"bytes_written":15430,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7825,"cfile_cache_miss":26,"cfile_cache_miss_bytes":22825,"cfile_init":6,"data dirs.queue_time_us":310,"data dirs.run_cpu_time_us":483,"data dirs.run_wall_time_us":949,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1011,"lbm_reads_lt_1ms":50,"lbm_write_time_us":37611,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":417} I0720 00:44:21.042271 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 4061 bytes on disk I0720 00:44:21.043020 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:21.044347 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000355 I0720 00:44:21.120890 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.087s user 0.020s sys 0.000s Metrics: {"bytes_written":9962,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":463,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"rows_written":204} I0720 00:44:21.122292 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 4482 bytes on disk I0720 00:44:21.123308 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":167,"lbm_reads_lt_1ms":8} I0720 00:44:21.124586 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=1.008165 W0720 00:44:21.164613 6648 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:41519, user_credentials={real_user=slave}} blocked reactor thread for 75399.2us I0720 00:44:21.132287 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.087s user 0.012s sys 0.004s Metrics: {"bytes_written":9111,"cfile_init":1,"data dirs.queue_time_us":38,"data dirs.run_cpu_time_us":99,"data dirs.run_wall_time_us":501,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":730,"lbm_writes_lt_1ms":23,"rows_written":171} I0720 00:44:21.185317 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 1767 bytes on disk I0720 00:44:21.186372 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4} I0720 00:44:21.189951 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2): perf score=1.013638 I0720 00:44:21.195911 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.577s user 0.029s sys 0.008s Metrics: {"bytes_written":14051,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6819,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19901,"cfile_init":6,"data dirs.queue_time_us":181645,"data dirs.run_cpu_time_us":796,"data dirs.run_wall_time_us":5871,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":82972,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":728,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":362,"thread_start_us":197907,"threads_started":3} I0720 00:44:21.202913 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 3538 bytes on disk I0720 00:44:21.206132 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:21.207515 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000689 I0720 00:44:21.232780 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.042s user 0.018s sys 0.010s Metrics: {"bytes_written":17118,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9027,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26261,"cfile_init":5,"data dirs.queue_time_us":341,"data dirs.run_cpu_time_us":559,"data dirs.run_wall_time_us":966,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1053,"lbm_reads_lt_1ms":46,"lbm_write_time_us":873,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":483} I0720 00:44:21.234520 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000394 I0720 00:44:21.254905 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.020s user 0.014s sys 0.002s Metrics: {"bytes_written":9086,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":86,"data dirs.run_wall_time_us":432,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":643,"lbm_writes_lt_1ms":23,"rows_written":170} I0720 00:44:21.257408 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 4670 bytes on disk I0720 00:44:21.274361 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.008s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":7767,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0720 00:44:21.275573 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=1.006584 I0720 00:44:21.396667 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.189s user 0.016s sys 0.010s Metrics: {"bytes_written":12837,"cfile_init":1,"data dirs.queue_time_us":54,"data dirs.run_cpu_time_us":124,"data dirs.run_wall_time_us":524,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21244,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":316} I0720 00:44:21.399108 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=1.005263 I0720 00:44:21.428653 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.106s user 0.021s sys 0.005s Metrics: {"bytes_written":16266,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8448,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24698,"cfile_init":6,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":567,"data dirs.run_wall_time_us":931,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1031,"lbm_reads_lt_1ms":50,"lbm_write_time_us":3535,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":451} I0720 00:44:21.430018 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000427 I0720 00:44:21.536839 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.107s user 0.018s sys 0.001s Metrics: {"bytes_written":10322,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":448,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"rows_written":217} I0720 00:44:21.538446 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 4376 bytes on disk I0720 00:44:21.539117 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":180,"lbm_reads_lt_1ms":4} I0720 00:44:21.540011 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.007809 I0720 00:44:21.569726 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.028s user 0.026s sys 0.001s Metrics: {"bytes_written":17348,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9195,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26847,"cfile_init":6,"data dirs.queue_time_us":309,"data dirs.run_cpu_time_us":481,"data dirs.run_wall_time_us":820,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":952,"lbm_reads_lt_1ms":50,"lbm_write_time_us":782,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":492} I0720 00:44:21.570996 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000452 I0720 00:44:21.590391 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.140s user 0.030s sys 0.000s Metrics: {"bytes_written":16774,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8808,"cfile_cache_miss":26,"cfile_cache_miss_bytes":25664,"cfile_init":7,"data dirs.queue_time_us":2313,"data dirs.run_cpu_time_us":603,"data dirs.run_wall_time_us":1034,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1788,"lbm_reads_lt_1ms":54,"lbm_write_time_us":841,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":471,"spinlock_wait_cycles":1280} I0720 00:44:21.592186 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000789 I0720 00:44:21.611449 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.040s user 0.016s sys 0.003s Metrics: {"bytes_written":9567,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":510,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":992,"lbm_writes_lt_1ms":23,"rows_written":190} I0720 00:44:21.614621 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=1.013557 I0720 00:44:21.676437 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.062s user 0.023s sys 0.006s Metrics: {"bytes_written":16908,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8886,"cfile_cache_miss":26,"cfile_cache_miss_bytes":25870,"cfile_init":6,"data dirs.queue_time_us":297,"data dirs.run_cpu_time_us":477,"data dirs.run_wall_time_us":839,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1151,"lbm_reads_lt_1ms":50,"lbm_write_time_us":1028,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":475} I0720 00:44:21.677781 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 4730 bytes on disk I0720 00:44:21.678802 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:21.679975 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000380 I0720 00:44:21.701757 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.021s user 0.011s sys 0.004s Metrics: {"bytes_written":9479,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":510,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":683,"lbm_writes_lt_1ms":23,"rows_written":185} I0720 00:44:21.703146 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 4610 bytes on disk I0720 00:44:21.703747 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4} I0720 00:44:21.704471 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 1891 bytes on disk I0720 00:44:21.705762 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":219,"lbm_reads_lt_1ms":4} I0720 00:44:21.706779 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=1.011879 W0720 00:44:21.723574 6648 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.131:38005, user_credentials={real_user=slave}} blocked reactor thread for 55852.2us I0720 00:44:21.727284 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.602s user 0.026s sys 0.004s Metrics: {"bytes_written":16123,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8335,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24383,"cfile_init":5,"data dirs.queue_time_us":1080,"data dirs.run_cpu_time_us":490,"data dirs.run_wall_time_us":1095,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1003,"lbm_reads_lt_1ms":46,"lbm_write_time_us":9154,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":445,"thread_start_us":5263,"threads_started":2} I0720 00:44:21.728642 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000562 I0720 00:44:21.741633 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.034s user 0.026s sys 0.003s Metrics: {"bytes_written":18745,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10217,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29715,"cfile_init":5,"data dirs.queue_time_us":423,"data dirs.run_cpu_time_us":552,"data dirs.run_wall_time_us":925,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":868,"lbm_reads_lt_1ms":46,"lbm_write_time_us":698,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":548} I0720 00:44:21.742545 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000304 I0720 00:44:21.757601 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.015s user 0.011s sys 0.000s Metrics: {"bytes_written":7759,"cfile_init":1,"data dirs.queue_time_us":40,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":536,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":881,"lbm_writes_lt_1ms":23,"rows_written":120} I0720 00:44:21.758563 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.008743 I0720 00:44:21.773222 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.044s user 0.016s sys 0.004s Metrics: {"bytes_written":10803,"cfile_init":1,"data dirs.queue_time_us":907,"data dirs.run_cpu_time_us":93,"data dirs.run_wall_time_us":491,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"rows_written":238} I0720 00:44:21.774395 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.132:0 --local_ip_for_outbound_sockets=127.0.58.132 --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:37529 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:21.774507 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(57d8956326d345eba0ced816663169f9): perf score=1.009503 I0720 00:44:21.787305 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.028s user 0.023s sys 0.005s Metrics: {"bytes_written":18392,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9976,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28836,"cfile_init":6,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":437,"data dirs.run_wall_time_us":799,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":968,"lbm_reads_lt_1ms":50,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":535} I0720 00:44:21.788326 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000300 I0720 00:44:21.804345 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.030s user 0.024s sys 0.004s Metrics: {"bytes_written":17170,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9121,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26671,"cfile_init":6,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":447,"data dirs.run_wall_time_us":843,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":973,"lbm_reads_lt_1ms":50,"lbm_write_time_us":1061,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":488} I0720 00:44:21.805567 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 4735 bytes on disk I0720 00:44:21.806135 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:21.807129 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 4330 bytes on disk I0720 00:44:21.807654 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:21.808415 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 4042 bytes on disk I0720 00:44:21.808918 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:21.809617 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 3789 bytes on disk I0720 00:44:21.810181 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:21.810957 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000575 I0720 00:44:21.812819 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.220s user 0.014s sys 0.008s Metrics: {"bytes_written":13307,"cfile_init":1,"data dirs.queue_time_us":811,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":504,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"rows_written":335} I0720 00:44:21.814194 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 4547 bytes on disk I0720 00:44:21.814769 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4} I0720 00:44:21.815613 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 3302 bytes on disk I0720 00:44:21.816166 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:21.816987 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 3200 bytes on disk I0720 00:44:21.817435 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:21.818245 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 2393 bytes on disk I0720 00:44:21.818686 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:21.819433 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=1.008994 I0720 00:44:21.820955 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.031s user 0.009s sys 0.006s Metrics: {"bytes_written":8087,"cfile_init":1,"data dirs.queue_time_us":49,"data dirs.run_cpu_time_us":94,"data dirs.run_wall_time_us":449,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"rows_written":132} I0720 00:44:21.822422 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 5343 bytes on disk I0720 00:44:21.824282 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:21.825055 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 5205 bytes on disk I0720 00:44:21.825569 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:21.826313 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=1.001364 I0720 00:44:21.833851 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.023s user 0.022s sys 0.000s Metrics: {"bytes_written":10814,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":453,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"rows_written":238} I0720 00:44:21.835045 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=1.013674 I0720 00:44:21.866036 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.031s user 0.029s sys 0.000s Metrics: {"bytes_written":17734,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9487,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27665,"cfile_init":6,"data dirs.queue_time_us":297,"data dirs.run_cpu_time_us":434,"data dirs.run_wall_time_us":794,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":885,"lbm_reads_lt_1ms":50,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":508} I0720 00:44:21.867451 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000584 I0720 00:44:21.879169 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.060s user 0.022s sys 0.006s Metrics: {"bytes_written":18313,"cfile_cache_hit":12,"cfile_cache_hit_bytes":11348,"cfile_cache_miss":24,"cfile_cache_miss_bytes":27234,"cfile_init":5,"data dirs.queue_time_us":9069,"data dirs.run_cpu_time_us":555,"data dirs.run_wall_time_us":980,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1512,"lbm_reads_lt_1ms":44,"lbm_write_time_us":684,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":530} I0720 00:44:21.880508 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 5163 bytes on disk I0720 00:44:21.881120 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:21.881961 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 2136 bytes on disk I0720 00:44:21.882469 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:21.883240 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000826 I0720 00:44:21.886452 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.060s user 0.030s sys 0.000s Metrics: {"bytes_written":18507,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10015,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29041,"cfile_init":6,"data dirs.queue_time_us":339,"data dirs.run_cpu_time_us":420,"data dirs.run_wall_time_us":832,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":914,"lbm_reads_lt_1ms":50,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":537} I0720 00:44:21.887624 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000307 I0720 00:44:21.889698 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.022s user 0.018s sys 0.003s Metrics: {"bytes_written":10801,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":89,"data dirs.run_wall_time_us":514,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":753,"lbm_writes_lt_1ms":23,"rows_written":238} I0720 00:44:21.890910 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 4932 bytes on disk I0720 00:44:21.891501 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:21.892282 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 2434 bytes on disk I0720 00:44:21.892772 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4} I0720 00:44:21.893471 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2): perf score=1.013482 I0720 00:44:21.912892 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.023s user 0.007s sys 0.005s Metrics: {"bytes_written":8069,"cfile_init":1,"data dirs.queue_time_us":2916,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":607,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"rows_written":130} I0720 00:44:21.919124 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 5204 bytes on disk I0720 00:44:21.921037 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:21.921862 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=1.005463 I0720 00:44:21.922312 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.039s user 0.019s sys 0.008s Metrics: {"bytes_written":14305,"cfile_init":1,"compiler_manager_pool.queue_time_us":22628,"compiler_manager_pool.run_cpu_time_us":27,"compiler_manager_pool.run_wall_time_us":27,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":471,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":667,"lbm_writes_lt_1ms":23,"rows_written":374} I0720 00:44:21.923651 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 5359 bytes on disk I0720 00:44:21.924408 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":155,"lbm_reads_lt_1ms":8} I0720 00:44:21.924957 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.031s user 0.019s sys 0.011s Metrics: {"bytes_written":19165,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10470,"cfile_cache_miss":26,"cfile_cache_miss_bytes":30454,"cfile_init":5,"data dirs.queue_time_us":770,"data dirs.run_cpu_time_us":494,"data dirs.run_wall_time_us":912,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":906,"lbm_reads_lt_1ms":46,"lbm_write_time_us":713,"lbm_writes_lt_1ms":23,"mutex_wait_us":14,"num_input_rowsets":2,"rows_written":562,"spinlock_wait_cycles":9984} I0720 00:44:21.925240 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=1.013482 I0720 00:44:21.926239 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=0.000565 I0720 00:44:21.950040 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.024s user 0.012s sys 0.008s Metrics: {"bytes_written":11824,"cfile_init":1,"compiler_manager_pool.queue_time_us":13358,"compiler_manager_pool.run_cpu_time_us":26,"compiler_manager_pool.run_wall_time_us":26,"data dirs.queue_time_us":73,"data dirs.run_cpu_time_us":140,"data dirs.run_wall_time_us":580,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"rows_written":277} I0720 00:44:21.951375 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9): perf score=1.011515 I0720 00:44:21.955024 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.030s user 0.028s sys 0.000s Metrics: {"bytes_written":18507,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10015,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29065,"cfile_init":5,"data dirs.queue_time_us":532,"data dirs.run_cpu_time_us":544,"data dirs.run_wall_time_us":922,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":917,"lbm_reads_lt_1ms":46,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"mutex_wait_us":24,"num_input_rowsets":2,"rows_written":537,"spinlock_wait_cycles":2816} I0720 00:44:21.956485 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 5204 bytes on disk I0720 00:44:21.957038 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:21.957756 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000774 I0720 00:44:21.966311 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.044s user 0.023s sys 0.000s Metrics: {"bytes_written":18353,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9908,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28888,"cfile_init":6,"data dirs.queue_time_us":2103,"data dirs.run_cpu_time_us":292,"data dirs.run_wall_time_us":638,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":935,"lbm_reads_lt_1ms":50,"lbm_write_time_us":4711,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":531} I0720 00:44:21.967557 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 5153 bytes on disk I0720 00:44:21.970511 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:44:21.971303 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000312 I0720 00:44:21.981453 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.030s user 0.028s sys 0.001s Metrics: {"bytes_written":18507,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10016,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29180,"cfile_init":6,"data dirs.queue_time_us":316,"data dirs.run_cpu_time_us":486,"data dirs.run_wall_time_us":805,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":858,"lbm_reads_lt_1ms":50,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":537} I0720 00:44:21.983033 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 5447 bytes on disk I0720 00:44:21.983147 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.025s user 0.022s sys 0.000s Metrics: {"bytes_written":13702,"cfile_init":1,"data dirs.queue_time_us":54,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":549,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"rows_written":350} I0720 00:44:21.983773 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:21.984467 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 3460 bytes on disk I0720 00:44:21.984563 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000557 I0720 00:44:21.985097 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4} I0720 00:44:21.985816 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2): perf score=1.012798 I0720 00:44:22.000473 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.029s user 0.009s sys 0.003s Metrics: {"bytes_written":7333,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":88,"data dirs.run_wall_time_us":466,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"rows_written":104} I0720 00:44:22.003130 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(57d8956326d345eba0ced816663169f9): perf score=1.004787 I0720 00:44:22.006322 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.022s user 0.018s sys 0.001s Metrics: {"bytes_written":10522,"cfile_init":1,"data dirs.queue_time_us":64,"data dirs.run_cpu_time_us":117,"data dirs.run_wall_time_us":480,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"rows_written":225} I0720 00:44:22.007536 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=1.009702 I0720 00:44:22.014822 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.029s user 0.023s sys 0.002s Metrics: {"bytes_written":19165,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10470,"cfile_cache_miss":26,"cfile_cache_miss_bytes":30408,"cfile_init":5,"data dirs.queue_time_us":548,"data dirs.run_cpu_time_us":578,"data dirs.run_wall_time_us":903,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":715,"lbm_reads_lt_1ms":46,"lbm_write_time_us":717,"lbm_writes_lt_1ms":23,"mutex_wait_us":16,"num_input_rowsets":2,"rows_written":562,"spinlock_wait_cycles":4608} I0720 00:44:22.016079 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000782 I0720 00:44:22.036864 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.021s user 0.016s sys 0.004s Metrics: {"bytes_written":12170,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":528,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":625,"lbm_writes_lt_1ms":23,"rows_written":289} I0720 00:44:22.038372 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 5447 bytes on disk I0720 00:44:22.038966 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:22.039716 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=1.013574 I0720 00:44:22.047086 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.039s user 0.026s sys 0.005s Metrics: {"bytes_written":18745,"cfile_cache_hit":12,"cfile_cache_hit_bytes":12533,"cfile_cache_miss":24,"cfile_cache_miss_bytes":27469,"cfile_init":6,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":449,"data dirs.run_wall_time_us":829,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":901,"lbm_reads_lt_1ms":48,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":548} I0720 00:44:22.048254 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000511 I0720 00:44:22.062726 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.059s user 0.021s sys 0.007s Metrics: {"bytes_written":17527,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9336,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27104,"cfile_init":6,"data dirs.queue_time_us":1056,"data dirs.run_cpu_time_us":507,"data dirs.run_wall_time_us":914,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":3739,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":842,"lbm_writes_lt_1ms":23,"mutex_wait_us":26,"num_input_rowsets":2,"rows_written":500} I0720 00:44:22.063946 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000309 I0720 00:44:22.067214 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.019s user 0.011s sys 0.004s Metrics: {"bytes_written":9126,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":536,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":599,"lbm_writes_lt_1ms":23,"rows_written":173} I0720 00:44:22.068457 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 5343 bytes on disk I0720 00:44:22.069030 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:22.069808 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9): 5204 bytes on disk I0720 00:44:22.069836 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.030s user 0.027s sys 0.000s Metrics: {"bytes_written":17922,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9633,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28097,"cfile_init":6,"data dirs.queue_time_us":331,"data dirs.run_cpu_time_us":476,"data dirs.run_wall_time_us":1025,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":851,"lbm_reads_lt_1ms":50,"lbm_write_time_us":703,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":516} I0720 00:44:22.070336 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(27478acfc8ee4027a84efa2c19ee11e9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4} I0720 00:44:22.071004 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 1789 bytes on disk I0720 00:44:22.071084 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000782 I0720 00:44:22.071508 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:22.072134 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=1.003514 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:22.080849 7106 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:22.081193 7106 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:22.084403 7106 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:22.084529 7106 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:22.084596 7106 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:22.084679 7106 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.132 I0720 00:44:22.092761 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.021s user 0.018s sys 0.000s Metrics: {"bytes_written":12072,"cfile_init":1,"data dirs.queue_time_us":69,"data dirs.run_cpu_time_us":140,"data dirs.run_wall_time_us":440,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1403,"lbm_writes_lt_1ms":23,"rows_written":286} I0720 00:44:22.094023 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.008239 I0720 00:44:22.094650 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.026s user 0.011s sys 0.002s Metrics: {"bytes_written":7461,"cfile_init":1,"data dirs.queue_time_us":75,"data dirs.run_cpu_time_us":136,"data dirs.run_wall_time_us":529,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"rows_written":108} I0720 00:44:22.095928 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 4847 bytes on disk I0720 00:44:22.096550 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4} I0720 00:44:22.097334 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=1.001479 I0720 00:44:22.097784 7106 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --webserver_interface=127.0.58.132 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:37529 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.132 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:22.108551 7106 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 92516us I0720 00:44:22.109449 7106 fs_manager.cc:263] Metadata directory not provided I0720 00:44:22.109573 7106 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:22.109982 7106 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:22.110158 7106 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:22.110872 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.038s user 0.029s sys 0.001s Metrics: {"bytes_written":18164,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9795,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28269,"cfile_init":5,"data dirs.queue_time_us":320,"data dirs.run_cpu_time_us":463,"data dirs.run_wall_time_us":962,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":912,"lbm_reads_lt_1ms":46,"lbm_write_time_us":749,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":525} I0720 00:44:22.112036 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000441 I0720 00:44:22.112957 7106 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/instance: uuid: "e1d7982f305c4a728a23a075e1e9ac7b" format_stamp: "Formatted at 2019-07-20 00:44:22 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:22.113658 7106 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-1/instance: uuid: "e1d7982f305c4a728a23a075e1e9ac7b" format_stamp: "Formatted at 2019-07-20 00:44:22 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:22.114181 7106 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-2/instance: uuid: "e1d7982f305c4a728a23a075e1e9ac7b" format_stamp: "Formatted at 2019-07-20 00:44:22 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:22.114658 7106 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/wal/instance: uuid: "e1d7982f305c4a728a23a075e1e9ac7b" format_stamp: "Formatted at 2019-07-20 00:44:22 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:22.125632 7106 fs_manager.cc:503] Time spent creating directory manager: real 0.011s user 0.005s sys 0.003s I0720 00:44:22.125898 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.032s user 0.021s sys 0.007s Metrics: {"bytes_written":18392,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9981,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29129,"cfile_init":6,"data dirs.queue_time_us":356,"data dirs.run_cpu_time_us":504,"data dirs.run_wall_time_us":901,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":934,"lbm_reads_lt_1ms":50,"lbm_write_time_us":740,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":535} I0720 00:44:22.127091 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 5205 bytes on disk I0720 00:44:22.127620 7106 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:22.127636 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:22.128316 7106 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:22.128397 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 4982 bytes on disk I0720 00:44:22.128842 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:22.129431 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000772 I0720 00:44:22.132004 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.020s user 0.015s sys 0.000s Metrics: {"bytes_written":9492,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":90,"data dirs.run_wall_time_us":486,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":617,"lbm_writes_lt_1ms":23,"rows_written":186} I0720 00:44:22.133105 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=1.012978 I0720 00:44:22.141058 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.043s user 0.023s sys 0.007s Metrics: {"bytes_written":18164,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9796,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28386,"cfile_init":6,"data dirs.queue_time_us":580,"data dirs.run_cpu_time_us":375,"data dirs.run_wall_time_us":736,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1306,"lbm_reads_lt_1ms":50,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":525} I0720 00:44:22.142360 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 5102 bytes on disk I0720 00:44:22.143600 7106 fs_manager.cc:419] Time spent opening block manager: real 0.013s user 0.002s sys 0.009s I0720 00:44:22.143601 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:22.143703 7106 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/wal uuid: "e1d7982f305c4a728a23a075e1e9ac7b" format_stamp: "Formatted at 2019-07-20 00:44:22 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:22.143918 7106 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:22.144363 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(d4520f8364cb4356bf859651b4a349b2): perf score=0.000281 I0720 00:44:22.152709 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.023s user 0.018s sys 0.000s Metrics: {"bytes_written":11774,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":98,"data dirs.run_wall_time_us":443,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2306,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":274} I0720 00:44:22.153908 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 2759 bytes on disk I0720 00:44:22.154611 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:22.155293 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=1.013798 I0720 00:44:22.156654 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.012s user 0.009s sys 0.000s Metrics: {"bytes_written":6705,"cfile_init":1,"data dirs.queue_time_us":60,"data dirs.run_cpu_time_us":106,"data dirs.run_wall_time_us":558,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":588,"lbm_writes_lt_1ms":23,"rows_written":79} I0720 00:44:22.157795 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2): perf score=1.008255 I0720 00:44:22.165347 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.032s user 0.028s sys 0.000s Metrics: {"bytes_written":17922,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9633,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28039,"cfile_init":6,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":489,"data dirs.run_wall_time_us":871,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1078,"lbm_reads_lt_1ms":50,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":516} I0720 00:44:22.166569 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 5102 bytes on disk I0720 00:44:22.167155 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:22.167834 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 4982 bytes on disk I0720 00:44:22.170104 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:22.170806 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000417 I0720 00:44:22.179303 7106 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:22.183296 7106 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:22.183429 7106 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:22.189644 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.034s user 0.026s sys 0.001s Metrics: {"bytes_written":17734,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9487,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27753,"cfile_init":5,"data dirs.queue_time_us":313,"data dirs.run_cpu_time_us":516,"data dirs.run_wall_time_us":898,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":859,"lbm_reads_lt_1ms":46,"lbm_write_time_us":2950,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"num_input_rowsets":2,"rows_written":508} I0720 00:44:22.190361 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.032s user 0.029s sys 0.000s Metrics: {"bytes_written":19165,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10470,"cfile_cache_miss":26,"cfile_cache_miss_bytes":30284,"cfile_init":6,"data dirs.queue_time_us":629,"data dirs.run_cpu_time_us":524,"data dirs.run_wall_time_us":950,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":844,"lbm_reads_lt_1ms":50,"lbm_write_time_us":2492,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"mutex_wait_us":2,"num_input_rowsets":2,"rows_written":562} I0720 00:44:22.190924 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 4932 bytes on disk I0720 00:44:22.191746 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2): 5447 bytes on disk I0720 00:44:22.192580 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(d4520f8364cb4356bf859651b4a349b2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:22.192605 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:22.194666 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000736 I0720 00:44:22.195346 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.024s user 0.011s sys 0.004s Metrics: {"bytes_written":8501,"cfile_init":1,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":509,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"rows_written":148} I0720 00:44:22.196189 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.008483 I0720 00:44:22.221056 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.026s user 0.021s sys 0.000s Metrics: {"bytes_written":11461,"cfile_init":1,"data dirs.queue_time_us":255,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":502,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"rows_written":264} I0720 00:44:22.223330 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 2641 bytes on disk I0720 00:44:22.224395 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:22.225107 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(57d8956326d345eba0ced816663169f9): perf score=1.009085 I0720 00:44:22.226177 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.030s user 0.027s sys 0.002s Metrics: {"bytes_written":18509,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10032,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29014,"cfile_init":6,"data dirs.queue_time_us":875,"data dirs.run_cpu_time_us":556,"data dirs.run_wall_time_us":909,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":939,"lbm_reads_lt_1ms":50,"lbm_write_time_us":720,"lbm_writes_lt_1ms":23,"mutex_wait_us":24,"num_input_rowsets":2,"rows_written":538} I0720 00:44:22.227056 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(79f36e06977e420f94ab39786832ecc9): perf score=0.000406 I0720 00:44:22.243223 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.016s user 0.013s sys 0.001s Metrics: {"bytes_written":7759,"cfile_init":1,"data dirs.queue_time_us":65,"data dirs.run_cpu_time_us":115,"data dirs.run_wall_time_us":554,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":610,"lbm_writes_lt_1ms":23,"rows_written":120} I0720 00:44:22.244246 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 5196 bytes on disk I0720 00:44:22.244818 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I0720 00:44:22.245479 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 1276 bytes on disk I0720 00:44:22.247822 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:22.248319 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9): perf score=1.008743 I0720 00:44:22.260768 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.035s user 0.019s sys 0.003s Metrics: {"bytes_written":17527,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9335,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27149,"cfile_init":5,"data dirs.queue_time_us":720,"data dirs.run_cpu_time_us":526,"data dirs.run_wall_time_us":843,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1585,"lbm_reads_lt_1ms":46,"lbm_write_time_us":567,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":500,"spinlock_wait_cycles":1920} I0720 00:44:22.261668 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031): perf score=0.000570 I0720 00:44:22.275779 7106 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.132:33507 I0720 00:44:22.276870 7106 webserver.cc:374] Webserver started at http://127.0.58.132:44453/ using document root and password file I0720 00:44:22.277483 7106 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb I0720 00:44:22.279532 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.018s user 0.012s sys 0.004s Metrics: {"bytes_written":9907,"cfile_init":1,"data dirs.queue_time_us":49,"data dirs.run_cpu_time_us":106,"data dirs.run_wall_time_us":466,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"rows_written":201} I0720 00:44:22.280614 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 7106 I0720 00:44:22.280619 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031): perf score=1.004402 I0720 00:44:22.282826 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.034s user 0.024s sys 0.007s Metrics: {"bytes_written":18392,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9976,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28836,"cfile_init":5,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":439,"data dirs.run_wall_time_us":803,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":934,"lbm_reads_lt_1ms":46,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":535} I0720 00:44:22.283812 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000323 I0720 00:44:22.298028 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.014s user 0.008s sys 0.003s Metrics: {"bytes_written":6897,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":567,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1219,"lbm_writes_lt_1ms":23,"rows_written":86} W0720 00:44:22.299345 6632 generic_service.cc:128] {username='slave'} at 127.0.0.1:48978 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:22.299543 6632 generic_service.cc:148] {username='slave'} at 127.0.0.1:48978 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:22.300987 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.994285 I0720 00:44:22.301121 7228 heartbeater.cc:324] Connected to a master server at 127.0.58.190:37529 I0720 00:44:22.301358 7228 heartbeater.cc:405] Registering TS with master... I0720 00:44:22.301968 7228 heartbeater.cc:454] Master 127.0.58.190:37529 requested a full tablet report, sending... I0720 00:44:22.303316 6468 ts_manager.cc:161] Registered new tserver with Master: e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507) I0720 00:44:22.304505 6468 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.132:55633 W0720 00:44:22.314169 6757 generic_service.cc:128] {username='slave'} at 127.0.0.1:34122 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:22.314303 6757 generic_service.cc:148] {username='slave'} at 127.0.0.1:34122 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:22.320276 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.039s user 0.028s sys 0.000s Metrics: {"bytes_written":18164,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9796,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28522,"cfile_init":6,"data dirs.queue_time_us":316,"data dirs.run_cpu_time_us":521,"data dirs.run_wall_time_us":931,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":933,"lbm_reads_lt_1ms":50,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":525} I0720 00:44:22.321406 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000480 W0720 00:44:22.324358 6881 generic_service.cc:128] {username='slave'} at 127.0.0.1:50612 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:22.324470 6881 generic_service.cc:148] {username='slave'} at 127.0.0.1:50612 changed flags via RPC: enable_tablet_copy from 'true' to 'false' W0720 00:44:22.338941 7225 generic_service.cc:128] {username='slave'} at 127.0.0.1:46250 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:22.339063 7225 generic_service.cc:148] {username='slave'} at 127.0.0.1:46250 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:22.340876 234 test_util.cc:212] Using random seed: 618232062 I0720 00:44:22.342011 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.034s user 0.028s sys 0.000s Metrics: {"bytes_written":18353,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9908,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28662,"cfile_init":6,"data dirs.queue_time_us":341,"data dirs.run_cpu_time_us":521,"data dirs.run_wall_time_us":1048,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":931,"lbm_reads_lt_1ms":50,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":531,"spinlock_wait_cycles":1408} I0720 00:44:22.343130 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9): 5205 bytes on disk I0720 00:44:22.343864 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(79f36e06977e420f94ab39786832ecc9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:22.344511 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 5153 bytes on disk I0720 00:44:22.344013 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.022s user 0.015s sys 0.001s Metrics: {"bytes_written":9185,"cfile_init":1,"data dirs.queue_time_us":49,"data dirs.run_cpu_time_us":101,"data dirs.run_wall_time_us":478,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"rows_written":176} I0720 00:44:22.345748 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031): 5102 bytes on disk I0720 00:44:22.346335 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(bf4081672af8427b9d5d11d8dfeae031) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:22.347069 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9): 4847 bytes on disk I0720 00:44:22.347546 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:22.348158 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 1798 bytes on disk I0720 00:44:22.348806 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:22.349789 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.008903 I0720 00:44:22.349820 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:22.361171 234 test_workload.cc:306] TestWorkload: Skipping table creation because table test-workload already exists W0720 00:44:22.362730 6632 generic_service.cc:128] {username='slave'} at 127.0.0.1:48978 forcing change of non-runtime-safe flag cfile_inject_corruption I0720 00:44:22.362872 6632 generic_service.cc:148] {username='slave'} at 127.0.0.1:48978 changed flags via RPC: cfile_inject_corruption from '0' to '1.0' I0720 00:44:22.368422 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.373915 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.379984 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.030s user 0.025s sys 0.004s Metrics: {"bytes_written":18509,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10032,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29070,"cfile_init":5,"data dirs.queue_time_us":681,"data dirs.run_cpu_time_us":494,"data dirs.run_wall_time_us":861,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":795,"lbm_reads_lt_1ms":46,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":538,"spinlock_wait_cycles":3456} I0720 00:44:22.380875 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 5196 bytes on disk I0720 00:44:22.381160 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.381376 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:44:22.389358 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.401742 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.422220 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.443938 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.000330 I0720 00:44:22.453336 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.009s user 0.003s sys 0.004s Metrics: {"bytes_written":6377,"cfile_init":1,"data dirs.queue_time_us":65,"data dirs.run_cpu_time_us":125,"data dirs.run_wall_time_us":447,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":462,"lbm_writes_lt_1ms":23,"rows_written":65} I0720 00:44:22.454185 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 745 bytes on disk I0720 00:44:22.454641 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} I0720 00:44:22.455154 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4): perf score=0.985334 I0720 00:44:22.458834 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.473044 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"bytes_written":17922,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9633,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27863,"cfile_init":5,"data dirs.queue_time_us":1069,"data dirs.run_cpu_time_us":478,"data dirs.run_wall_time_us":857,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":597,"lbm_reads_lt_1ms":46,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"mutex_wait_us":118,"num_input_rowsets":2,"rows_written":516,"spinlock_wait_cycles":12032} I0720 00:44:22.473835 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4): 4982 bytes on disk I0720 00:44:22.474275 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(52408e87c8e54a7394a12dc191a9b4a4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:44:22.527719 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.632498 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.000343 I0720 00:44:22.641571 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.009s user 0.003s sys 0.004s Metrics: {"bytes_written":6225,"cfile_init":1,"data dirs.queue_time_us":67,"data dirs.run_cpu_time_us":106,"data dirs.run_wall_time_us":492,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":500,"lbm_writes_lt_1ms":23,"rows_written":60} I0720 00:44:22.642402 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 704 bytes on disk I0720 00:44:22.642938 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:22.643579 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c): perf score=0.978910 I0720 00:44:22.660426 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.661986 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.018s user 0.018s sys 0.000s Metrics: {"bytes_written":18353,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9903,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28567,"cfile_init":5,"data dirs.queue_time_us":746,"data dirs.run_cpu_time_us":414,"data dirs.run_wall_time_us":822,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":542,"lbm_reads_lt_1ms":46,"lbm_write_time_us":478,"lbm_writes_lt_1ms":23,"mutex_wait_us":23,"num_input_rowsets":2,"rows_written":531,"spinlock_wait_cycles":12032} I0720 00:44:22.662788 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c): 5153 bytes on disk I0720 00:44:22.663252 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(49ad7f72eaa04a14bc64fa4f6cbce73c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} I0720 00:44:22.725239 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(7139dc2492b147989250601c720f324c): perf score=0.000330 I0720 00:44:22.733404 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.008s user 0.006s sys 0.000s Metrics: {"bytes_written":5849,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":132,"data dirs.run_wall_time_us":520,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":453,"lbm_writes_lt_1ms":23,"rows_written":46} I0720 00:44:22.734239 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 573 bytes on disk I0720 00:44:22.734722 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:22.735189 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(7139dc2492b147989250601c720f324c): perf score=1.006847 I0720 00:44:22.753059 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.018s user 0.014s sys 0.004s Metrics: {"bytes_written":18509,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10032,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28938,"cfile_init":5,"data dirs.queue_time_us":614,"data dirs.run_cpu_time_us":320,"data dirs.run_wall_time_us":706,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":561,"lbm_reads_lt_1ms":46,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":538,"spinlock_wait_cycles":23808} I0720 00:44:22.753937 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.000317 I0720 00:44:22.760946 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: FlushMRSOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":5544,"cfile_init":1,"data dirs.queue_time_us":59,"data dirs.run_cpu_time_us":97,"data dirs.run_wall_time_us":431,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":378,"lbm_writes_lt_1ms":23,"rows_written":33,"spinlock_wait_cycles":20736,"thread_start_us":160,"threads_started":1,"wal-append.queue_time_us":245} I0720 00:44:22.761708 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c): 5196 bytes on disk I0720 00:44:22.762343 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(7139dc2492b147989250601c720f324c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:22.762883 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007): perf score=0.992376 I0720 00:44:22.782197 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: CompactRowSetsOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.019s user 0.016s sys 0.003s Metrics: {"bytes_written":17734,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9487,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27485,"cfile_init":6,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":478,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":651,"lbm_reads_lt_1ms":50,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":508} I0720 00:44:22.782949 6885 maintenance_manager.cc:327] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Scheduling UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007): 4932 bytes on disk I0720 00:44:22.783347 6779 maintenance_manager.cc:532] P 5aa5a77f1d2c4510a9a7e49bc2bb32aa: UndoDeltaBlockGCOp(a5154a3c09a5435fb2ac44744e8e7007) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4} W0720 00:44:22.851052 6547 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:44:22.851114 6547 fault_injection.cc:45] THIS SERVER MAY CRASH! E0720 00:44:22.851239 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000164 I0720 00:44:22.851377 6547 ts_tablet_manager.cc:1475] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:22.851734 6636 maintenance_manager.cc:327] P b88011e9e0d24f79882735c95c8fc275: Scheduling FlushMRSOp(57d8956326d345eba0ced816663169f9): perf score=0.000299 W0720 00:44:22.851969 6530 tablet_replica_mm_ops.cc:201] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: failed to flush MRS: Illegal state: Tablet has been stopped I0720 00:44:22.852071 6530 maintenance_manager.cc:532] P b88011e9e0d24f79882735c95c8fc275: FlushMRSOp(57d8956326d345eba0ced816663169f9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} W0720 00:44:22.851923 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "57d8956326d345eba0ced816663169f9" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437862500012032 propagated_timestamp: 6404437863579475968 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000164 at offset=1661 size=24: Checksum does not match: 234513494 vs expected 234513494 I0720 00:44:22.852336 7246 tablet_replica.cc:273] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica I0720 00:44:22.853050 7246 raft_consensus.cc:2147] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:22.853273 7246 raft_consensus.cc:2174] T 57d8956326d345eba0ced816663169f9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:22.914417 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling FlushMRSOp(f49376a672014e499d7fc53f9325c6f6): perf score=0.000306 I0720 00:44:22.921270 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:22.922796 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: FlushMRSOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.008s user 0.007s sys 0.000s Metrics: {"bytes_written":5122,"cfile_init":1,"data dirs.queue_time_us":75,"data dirs.run_cpu_time_us":157,"data dirs.run_wall_time_us":476,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":450,"lbm_writes_lt_1ms":23,"rows_written":18,"thread_start_us":213,"threads_started":1,"wal-append.queue_time_us":327} I0720 00:44:22.923704 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 310 bytes on disk I0720 00:44:22.924248 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4} I0720 00:44:22.924763 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6): perf score=1.003174 I0720 00:44:22.942821 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: CompactRowSetsOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.018s user 0.017s sys 0.000s Metrics: {"bytes_written":18745,"cfile_cache_hit":20,"cfile_cache_hit_bytes":28266,"cfile_cache_miss":16,"cfile_cache_miss_bytes":11522,"cfile_init":3,"data dirs.queue_time_us":371,"data dirs.run_cpu_time_us":259,"data dirs.run_wall_time_us":641,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":362,"lbm_reads_lt_1ms":28,"lbm_write_time_us":474,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":548,"spinlock_wait_cycles":1792} I0720 00:44:22.943493 6761 maintenance_manager.cc:327] P 29cb3a99090f4ceeb99efc07786b6426: Scheduling UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6): 5343 bytes on disk I0720 00:44:22.943912 6655 maintenance_manager.cc:532] P 29cb3a99090f4ceeb99efc07786b6426: UndoDeltaBlockGCOp(f49376a672014e499d7fc53f9325c6f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4} E0720 00:44:22.974692 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000260 I0720 00:44:22.974781 6547 ts_tablet_manager.cc:1475] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:22.975025 7246 tablet_replica.cc:273] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica I0720 00:44:22.975188 7246 raft_consensus.cc:2147] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus shutting down. W0720 00:44:22.974987 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "79f36e06977e420f94ab39786832ecc9" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437862500012032 propagated_timestamp: 6404437863929675776 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000260 at offset=1810 size=24: Checksum does not match: 2270883426 vs expected 2270883426 I0720 00:44:22.975317 7246 raft_consensus.cc:2174] T 79f36e06977e420f94ab39786832ecc9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! E0720 00:44:22.989233 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000200 I0720 00:44:22.989300 6547 ts_tablet_manager.cc:1475] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:22.989531 7246 tablet_replica.cc:273] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:22.989498 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "27478acfc8ee4027a84efa2c19ee11e9" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437862500012032 propagated_timestamp: 6404437864392798208 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000200 at offset=1864 size=24: Checksum does not match: 1926170819 vs expected 1926170819 I0720 00:44:22.989742 7246 raft_consensus.cc:2147] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:22.989928 7246 raft_consensus.cc:2174] T 27478acfc8ee4027a84efa2c19ee11e9 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:23.284301 6648 consensus_queue.cc:544] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:23.284462 6648 consensus_peers.cc:458] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.286675 6468 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 27478acfc8ee4027a84efa2c19ee11e9 (attempt 1) I0720 00:44:23.288239 6717 consensus_queue.cc:206] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:23.289906 6840 raft_consensus.cc:1184] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch) I0720 00:44:23.290391 7100 consensus_queue.cc:984] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:23.292449 7248 raft_consensus.cc:2792] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.292866 6840 raft_consensus.cc:2792] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.295192 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 27478acfc8ee4027a84efa2c19ee11e9 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:23.296573 6468 catalog_manager.cc:4067] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.300560 6465 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 27478acfc8ee4027a84efa2c19ee11e9 (attempt 1) I0720 00:44:23.301790 6717 consensus_queue.cc:206] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.108, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:23.303498 6840 raft_consensus.cc:1184] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 109. (index mismatch) I0720 00:44:23.304028 7248 consensus_queue.cc:984] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s I0720 00:44:23.306460 7248 raft_consensus.cc:2792] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.307771 6840 raft_consensus.cc:2792] T 27478acfc8ee4027a84efa2c19ee11e9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.309553 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27478acfc8ee4027a84efa2c19ee11e9 with cas_config_opid_index 108: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:23.309994 6465 catalog_manager.cc:4067] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:23.310297 7228 heartbeater.cc:446] Master 127.0.58.190:37529 was elected leader, sending a full tablet report... W0720 00:44:23.317590 6651 consensus_peers.cc:458] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27478acfc8ee4027a84efa2c19ee11e9. This is attempt 1: this message will repeat every 5th retry. E0720 00:44:23.354351 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000236 I0720 00:44:23.354441 6547 ts_tablet_manager.cc:1475] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:23.354683 7246 tablet_replica.cc:273] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:23.354629 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437865074429952 propagated_timestamp: 6404437865887531008 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000236 at offset=1753 size=24: Checksum does not match: 3431501168 vs expected 3431501168 I0720 00:44:23.354836 7246 raft_consensus.cc:2147] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:23.355177 7246 raft_consensus.cc:2174] T 52408e87c8e54a7394a12dc191a9b4a4 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! E0720 00:44:23.363765 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000272 I0720 00:44:23.363824 6547 ts_tablet_manager.cc:1475] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:23.364043 7246 tablet_replica.cc:273] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:23.363987 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "49ad7f72eaa04a14bc64fa4f6cbce73c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437865074429952 propagated_timestamp: 6404437865927655424 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000272 at offset=1840 size=24: Checksum does not match: 2332407364 vs expected 2332407364 I0720 00:44:23.364166 7246 raft_consensus.cc:2147] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:23.364298 7246 raft_consensus.cc:2174] T 49ad7f72eaa04a14bc64fa4f6cbce73c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! E0720 00:44:23.384423 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000188 I0720 00:44:23.384501 6547 ts_tablet_manager.cc:1475] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:23.384721 7246 tablet_replica.cc:273] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:23.384665 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "d4520f8364cb4356bf859651b4a349b2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437865074429952 propagated_timestamp: 6404437866012950528 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000188 at offset=1957 size=24: Checksum does not match: 1280897905 vs expected 1280897905 I0720 00:44:23.384862 7246 raft_consensus.cc:2147] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:23.385069 7246 raft_consensus.cc:2174] T d4520f8364cb4356bf859651b4a349b2 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:23.420168 6648 consensus_queue.cc:544] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:23.420298 6648 consensus_peers.cc:458] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.421965 6470 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 57d8956326d345eba0ced816663169f9 (attempt 1) I0720 00:44:23.422937 6717 consensus_queue.cc:206] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:23.424479 6840 raft_consensus.cc:1184] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:23.424935 7100 consensus_queue.cc:984] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:23.427021 7248 raft_consensus.cc:2792] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.427469 6840 raft_consensus.cc:2792] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.429311 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 57d8956326d345eba0ced816663169f9 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:23.430034 6470 catalog_manager.cc:4067] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index -1 to 106, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:23.432561 6470 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 57d8956326d345eba0ced816663169f9 (attempt 1) I0720 00:44:23.433593 6717 consensus_queue.cc:206] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:23.435128 6840 raft_consensus.cc:1184] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) W0720 00:44:23.435400 6651 consensus_peers.cc:458] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57d8956326d345eba0ced816663169f9. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.435741 7100 consensus_queue.cc:984] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:23.437356 7248 raft_consensus.cc:2792] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.437875 6840 raft_consensus.cc:2792] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.439712 234 disk_failure-itest.cc:256] Currently has 6 failed tablets I0720 00:44:23.440093 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 57d8956326d345eba0ced816663169f9 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:23.441184 6470 catalog_manager.cc:4067] T 57d8956326d345eba0ced816663169f9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index 106 to 107, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.526743 6772 consensus_queue.cc:544] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:23.526859 6772 consensus_peers.cc:458] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.528735 6465 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 79f36e06977e420f94ab39786832ecc9 (attempt 1) I0720 00:44:23.530122 6840 consensus_queue.cc:206] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:23.531790 6717 raft_consensus.cc:1184] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:23.532219 6928 consensus_queue.cc:984] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:23.534226 7260 raft_consensus.cc:2792] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index -1 to 107, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.534734 6717 raft_consensus.cc:2792] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index -1 to 107, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.536777 6451 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 79f36e06977e420f94ab39786832ecc9 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:23.537968 6465 catalog_manager.cc:4067] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index -1 to 107, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:23.540443 6465 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 79f36e06977e420f94ab39786832ecc9 (attempt 1) I0720 00:44:23.541488 6840 consensus_queue.cc:206] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:23.543368 6717 raft_consensus.cc:1184] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch) I0720 00:44:23.543740 7260 consensus_queue.cc:984] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:23.545523 6928 raft_consensus.cc:2792] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 107 to 108, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.546007 6717 raft_consensus.cc:2792] T 79f36e06977e420f94ab39786832ecc9 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 107 to 108, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.547724 6451 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 79f36e06977e420f94ab39786832ecc9 with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:23.549216 6465 catalog_manager.cc:4067] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index 107 to 108, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0720 00:44:23.549664 6775 consensus_peers.cc:458] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79f36e06977e420f94ab39786832ecc9. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.747241 6648 consensus_queue.cc:544] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:23.747380 6648 consensus_peers.cc:458] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.749119 6470 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 49ad7f72eaa04a14bc64fa4f6cbce73c (attempt 1) I0720 00:44:23.750146 6717 consensus_queue.cc:206] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:23.751758 6840 raft_consensus.cc:1184] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch) I0720 00:44:23.752183 7248 consensus_queue.cc:984] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:23.754228 7259 raft_consensus.cc:2792] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.754670 6840 raft_consensus.cc:2792] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:23.757088 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 49ad7f72eaa04a14bc64fa4f6cbce73c with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:23.758147 6470 catalog_manager.cc:4067] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:23.760505 6470 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 49ad7f72eaa04a14bc64fa4f6cbce73c (attempt 1) I0720 00:44:23.761636 6717 consensus_queue.cc:206] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.108, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:23.763164 6840 raft_consensus.cc:1184] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 109. (index mismatch) I0720 00:44:23.763540 7259 consensus_queue.cc:984] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s W0720 00:44:23.763912 6651 consensus_peers.cc:458] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49ad7f72eaa04a14bc64fa4f6cbce73c. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:23.765094 7100 raft_consensus.cc:2792] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.765606 6840 raft_consensus.cc:2792] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:23.767303 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 49ad7f72eaa04a14bc64fa4f6cbce73c with cas_config_opid_index 108: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:23.768425 6470 catalog_manager.cc:4067] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0720 00:44:23.894376 7259 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:23.926398 7259 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:24.008446 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:24.213306 7259 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:24.387817 6800 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "d4520f8364cb4356bf859651b4a349b2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437865074429952 propagated_timestamp: 6404437866012950528 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1563583463152937 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1563583463130701 usec, L: 0 Physical time difference: 0.022s I0720 00:44:24.388350 6800 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:50638 (request call id 11) took 1001ms. Request Metrics: {} W0720 00:44:24.423398 7259 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:24.424474 7259 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:24.427096 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:24.444329 234 disk_failure-itest.cc:256] Currently has 6 failed tablets I0720 00:44:24.578166 7260 raft_consensus.cc:472] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:24.578287 7260 raft_consensus.cc:494] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.578837 7260 leader_election.cc:296] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) I0720 00:44:24.579218 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:24.579320 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:24.579471 6717 raft_consensus.cc:2311] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:24.580020 6772 leader_election.cc:348] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:24.580099 6772 leader_election.cc:310] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275 I0720 00:44:24.580310 7260 raft_consensus.cc:2592] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0720 00:44:24.632478 7260 raft_consensus.cc:472] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:24.632604 7260 raft_consensus.cc:494] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.633096 7260 leader_election.cc:296] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) I0720 00:44:24.633622 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:24.633626 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:24.633870 6717 raft_consensus.cc:2311] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:24.634181 6772 leader_election.cc:348] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:24.634315 6774 leader_election.cc:310] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275 I0720 00:44:24.634536 7260 raft_consensus.cc:2592] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:24.669865 7259 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:24.764858 7259 raft_consensus.cc:472] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:24.764991 7259 raft_consensus.cc:494] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.765542 7259 leader_election.cc:296] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:24.766119 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:24.766145 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:24.766328 6840 raft_consensus.cc:2350] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. W0720 00:44:24.766619 6648 leader_election.cc:348] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:24.766811 6648 leader_election.cc:310] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:24.767029 7259 raft_consensus.cc:2644] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:24.767102 7259 raft_consensus.cc:472] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:24.767159 7259 raft_consensus.cc:2890] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:24.768923 7259 raft_consensus.cc:494] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.769317 7259 leader_election.cc:296] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:24.769737 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:24.769760 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:24.769883 6840 raft_consensus.cc:2890] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:24.770191 6648 leader_election.cc:348] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:24.771540 6840 raft_consensus.cc:2350] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 2. I0720 00:44:24.772539 6648 leader_election.cc:310] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:24.772796 7259 raft_consensus.cc:2644] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:24.773083 7259 raft_consensus.cc:667] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:24.773269 7259 consensus_queue.cc:206] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.775334 6465 catalog_manager.cc:4067] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 1 to 2, leader changed from b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } I0720 00:44:24.861096 7259 raft_consensus.cc:472] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:24.861259 7259 raft_consensus.cc:494] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.861814 7259 leader_election.cc:296] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:24.862406 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:24.862424 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:24.862576 6840 raft_consensus.cc:2350] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. W0720 00:44:24.862915 6648 leader_election.cc:348] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:24.863104 6648 leader_election.cc:310] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:24.863368 7259 raft_consensus.cc:2644] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:24.863423 7259 raft_consensus.cc:472] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:24.863487 7259 raft_consensus.cc:2890] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:24.865011 7259 raft_consensus.cc:494] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.865435 7259 leader_election.cc:296] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:24.865870 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:24.866004 6840 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "d4520f8364cb4356bf859651b4a349b2" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:24.866153 6840 raft_consensus.cc:2890] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:24.866376 6648 leader_election.cc:348] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:24.868042 6840 raft_consensus.cc:2350] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 2. I0720 00:44:24.868428 6648 leader_election.cc:310] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:24.868750 7259 raft_consensus.cc:2644] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:24.868862 7259 raft_consensus.cc:667] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:24.869045 7259 consensus_queue.cc:206] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:24.871469 6465 catalog_manager.cc:4067] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 1 to 2, leader changed from b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } W0720 00:44:24.925262 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:24.966854 7259 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.023530 7259 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.025246 6800 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "52408e87c8e54a7394a12dc191a9b4a4" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437870925254656 propagated_timestamp: 6404437872736419840 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.948s secs, (max is 0.750s): W0720 00:44:25.048599 6800 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "d4520f8364cb4356bf859651b4a349b2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437870925254656 propagated_timestamp: 6404437872832663552 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.917s secs, (max is 0.750s): I0720 00:44:25.184365 6840 raft_consensus.cc:1184] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:25.184792 7259 consensus_queue.cc:984] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:25.188387 6648 consensus_queue.cc:544] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:25.188464 6648 consensus_peers.cc:458] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:25.191735 6465 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 52408e87c8e54a7394a12dc191a9b4a4 (attempt 1) I0720 00:44:25.192589 6717 consensus_queue.cc:206] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:25.193660 6840 raft_consensus.cc:1184] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:25.194007 7266 consensus_queue.cc:984] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:25.195475 7275 raft_consensus.cc:2792] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:25.195945 6840 raft_consensus.cc:2792] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:25.197811 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 52408e87c8e54a7394a12dc191a9b4a4 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:25.198601 6465 catalog_manager.cc:4067] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index -1 to 107, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:25.201067 6465 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 52408e87c8e54a7394a12dc191a9b4a4 (attempt 1) I0720 00:44:25.201997 6717 consensus_queue.cc:206] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:25.203552 6840 raft_consensus.cc:1184] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) W0720 00:44:25.203758 6651 consensus_peers.cc:458] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52408e87c8e54a7394a12dc191a9b4a4. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:25.203961 7275 consensus_queue.cc:984] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:25.205386 7266 raft_consensus.cc:2792] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:25.205899 6840 raft_consensus.cc:2792] T 52408e87c8e54a7394a12dc191a9b4a4 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:25.207912 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 52408e87c8e54a7394a12dc191a9b4a4 with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:25.209120 6465 catalog_manager.cc:4067] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index 107 to 108, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } E0720 00:44:25.229549 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000176 I0720 00:44:25.229610 6547 ts_tablet_manager.cc:1475] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:25.230103 7277 tablet_replica.cc:273] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:25.229965 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437873559752704 propagated_timestamp: 6404437873568718848 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000176 at offset=1759 size=24: Checksum does not match: 829883413 vs expected 829883413 W0720 00:44:25.230273 7275 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:25.230271 7277 raft_consensus.cc:2147] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:25.230515 7277 raft_consensus.cc:2174] T a5154a3c09a5435fb2ac44744e8e7007 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:25.375798 6840 raft_consensus.cc:1184] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) I0720 00:44:25.376296 7275 consensus_queue.cc:984] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:25.379590 6648 consensus_queue.cc:544] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:25.379895 6648 consensus_peers.cc:458] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:25.381945 6470 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet d4520f8364cb4356bf859651b4a349b2 (attempt 1) I0720 00:44:25.382840 6717 consensus_queue.cc:206] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 2.108, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:25.383924 6840 raft_consensus.cc:1184] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 2 index: 108. Preceding OpId from leader: term: 2 index: 109. (index mismatch) I0720 00:44:25.384312 7266 consensus_queue.cc:984] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s I0720 00:44:25.385934 7275 raft_consensus.cc:2792] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Committing config change with OpId 2.109: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:25.386448 6839 raft_consensus.cc:2792] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Committing config change with OpId 2.109: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:25.388854 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet d4520f8364cb4356bf859651b4a349b2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:25.389422 6470 catalog_manager.cc:4067] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:25.391909 6470 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet d4520f8364cb4356bf859651b4a349b2 (attempt 1) I0720 00:44:25.392987 6717 consensus_queue.cc:206] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 2.109, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:25.393702 6470 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet d4520f8364cb4356bf859651b4a349b2 (attempt 1) I0720 00:44:25.394295 6839 raft_consensus.cc:1184] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 2 index: 109. Preceding OpId from leader: term: 2 index: 110. (index mismatch) W0720 00:44:25.394757 6651 consensus_peers.cc:458] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4520f8364cb4356bf859651b4a349b2. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:25.394924 7266 consensus_queue.cc:984] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s I0720 00:44:25.395428 6453 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4520f8364cb4356bf859651b4a349b2 with cas_config_opid_index 109: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } }. Pending config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:25.395530 6453 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4520f8364cb4356bf859651b4a349b2 with cas_config_opid_index 109 with a delay of 32 ms (attempt = 1) I0720 00:44:25.396453 7259 raft_consensus.cc:2792] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Committing config change with OpId 2.110: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:25.396952 6839 raft_consensus.cc:2792] T d4520f8364cb4356bf859651b4a349b2 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Committing config change with OpId 2.110: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:25.399327 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4520f8364cb4356bf859651b4a349b2 with cas_config_opid_index 109: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:25.399814 6470 catalog_manager.cc:4067] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:25.428795 6453 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d4520f8364cb4356bf859651b4a349b2 with cas_config_opid_index 109: aborting the task: latest config opid_index 110; task opid_index 109 W0720 00:44:25.448712 7266 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:25.449276 234 disk_failure-itest.cc:256] Currently has 7 failed tablets W0720 00:44:25.452822 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.480336 7266 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.578336 7266 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.729599 7266 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.793221 7266 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.914137 7266 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.975014 7266 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:25.979842 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.117341 7266 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.127692 7266 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.233760 6669 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437873559752704 propagated_timestamp: 6404437873568718848 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1563583465224549 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1563583464937375 usec, L: 0 Physical time difference: 0.287s I0720 00:44:26.234398 6669 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:34144 (request call id 13) took 1001ms. Request Metrics: {} W0720 00:44:26.237267 7266 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.264860 6800 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437873559752704 propagated_timestamp: 6404437873568718848 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.138s secs, (max is 0.750s): W0720 00:44:26.323379 6800 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437873559752704 propagated_timestamp: 6404437873568718848 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.196s secs, (max is 0.750s): I0720 00:44:26.438714 7266 raft_consensus.cc:472] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:26.438854 7266 raft_consensus.cc:494] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:26.439405 7266 leader_election.cc:296] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:26.439958 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:26.439958 6839 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:26.440177 6839 raft_consensus.cc:2311] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29cb3a99090f4ceeb99efc07786b6426 for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:26.440472 6648 leader_election.cc:348] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:26.440691 6648 leader_election.cc:310] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426; no voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, b88011e9e0d24f79882735c95c8fc275 I0720 00:44:26.440918 7266 raft_consensus.cc:2592] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0720 00:44:26.454205 234 disk_failure-itest.cc:256] Currently has 7 failed tablets W0720 00:44:26.461032 7266 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.465890 7266 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.469739 6669 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437873559752704 propagated_timestamp: 6404437873568718848 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.531s secs, (max is 0.750s): W0720 00:44:26.497792 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:26.628159 7260 raft_consensus.cc:472] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:26.628302 7260 raft_consensus.cc:494] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:26.628876 7260 leader_election.cc:296] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519) I0720 00:44:26.629472 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:26.629500 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:26.629637 6717 raft_consensus.cc:2350] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 1. I0720 00:44:26.630029 6774 leader_election.cc:310] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: W0720 00:44:26.630161 6772 leader_election.cc:348] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:26.630288 7260 raft_consensus.cc:2644] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:26.630331 7260 raft_consensus.cc:472] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:26.630373 7260 raft_consensus.cc:2890] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:26.631837 7260 raft_consensus.cc:494] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:26.632197 7260 leader_election.cc:296] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 election: Requested vote from peers 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329), b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519) I0720 00:44:26.632592 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" I0720 00:44:26.632721 6717 raft_consensus.cc:2890] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:26.632695 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" W0720 00:44:26.633092 6772 leader_election.cc:348] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:26.634196 6717 raft_consensus.cc:2350] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa in term 2. I0720 00:44:26.634475 6774 leader_election.cc:310] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:26.634681 7260 raft_consensus.cc:2644] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:26.634982 7260 raft_consensus.cc:667] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 LEADER]: Becoming Leader. State: Replica: 5aa5a77f1d2c4510a9a7e49bc2bb32aa, State: Running, Role: LEADER I0720 00:44:26.635157 7260 consensus_queue.cc:206] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:26.637218 6465 catalog_manager.cc:4067] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: term changed from 1 to 2, leader changed from b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) to 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131). New cstate: current_term: 2 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } W0720 00:44:26.659831 7266 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.708516 7266 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.713757 7266 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.760439 6669 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "a5154a3c09a5435fb2ac44744e8e7007" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437873559752704 propagated_timestamp: 6404437873568718848 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.822s secs, (max is 0.750s): W0720 00:44:26.895618 7266 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:26.975870 7260 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:27.053534 7266 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:27.060727 7266 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:27.083906 6772 consensus_queue.cc:544] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:27.084022 6772 consensus_peers.cc:458] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:27.254276 6717 raft_consensus.cc:1184] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) I0720 00:44:27.254772 7279 consensus_queue.cc:984] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:27.259776 6465 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet a5154a3c09a5435fb2ac44744e8e7007 (attempt 1) I0720 00:44:27.260696 6839 consensus_queue.cc:206] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 2.108, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:27.261781 6717 raft_consensus.cc:1184] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 108. Preceding OpId from leader: term: 2 index: 109. (index mismatch) I0720 00:44:27.262167 7279 consensus_queue.cc:984] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s I0720 00:44:27.263659 7260 raft_consensus.cc:2792] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 LEADER]: Committing config change with OpId 2.109: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.264098 6717 raft_consensus.cc:2792] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Committing config change with OpId 2.109: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.265888 6451 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet a5154a3c09a5435fb2ac44744e8e7007 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:27.266783 6465 catalog_manager.cc:4067] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:27.269666 6465 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a5154a3c09a5435fb2ac44744e8e7007 (attempt 1) I0720 00:44:27.270669 6839 consensus_queue.cc:206] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 2.109, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:27.271930 6717 raft_consensus.cc:1184] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 109. Preceding OpId from leader: term: 2 index: 110. (index mismatch) W0720 00:44:27.272328 6775 consensus_peers.cc:458] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5154a3c09a5435fb2ac44744e8e7007. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:27.272346 7279 consensus_queue.cc:984] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s I0720 00:44:27.273821 7260 raft_consensus.cc:2792] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 LEADER]: Committing config change with OpId 2.110: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:27.274327 6717 raft_consensus.cc:2792] T a5154a3c09a5435fb2ac44744e8e7007 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Committing config change with OpId 2.110: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } W0720 00:44:27.275370 7266 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:27.276407 6451 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a5154a3c09a5435fb2ac44744e8e7007 with cas_config_opid_index 109: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:27.277386 6465 catalog_manager.cc:4067] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0720 00:44:27.329754 7266 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:27.355329 7266 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled E0720 00:44:27.400667 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000212 I0720 00:44:27.400749 6547 ts_tablet_manager.cc:1475] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:27.401989 7292 tablet_replica.cc:273] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:27.401155 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "f49376a672014e499d7fc53f9325c6f6" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 6404437881437470720 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000212 at offset=1859 size=24: Checksum does not match: 623161563 vs expected 623161563 I0720 00:44:27.402168 7292 raft_consensus.cc:2147] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:27.402312 7292 raft_consensus.cc:2174] T f49376a672014e499d7fc53f9325c6f6 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:27.458940 234 disk_failure-itest.cc:256] Currently has 8 failed tablets E0720 00:44:27.486626 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000248 I0720 00:44:27.486698 6547 ts_tablet_manager.cc:1475] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:27.486924 7292 tablet_replica.cc:273] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica I0720 00:44:27.487123 7292 raft_consensus.cc:2147] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus shutting down. W0720 00:44:27.486902 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "7139dc2492b147989250601c720f324c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437882495889408 propagated_timestamp: 6404437882813403136 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000248 at offset=1849 size=24: Checksum does not match: 1960879312 vs expected 1960879312 I0720 00:44:27.487272 7292 raft_consensus.cc:2174] T 7139dc2492b147989250601c720f324c P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! W0720 00:44:27.559794 7266 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:27.560871 7279 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:27.623073 7266 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:27.654037 7266 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled E0720 00:44:27.711385 6547 cfile_reader.cc:594] Encountered corrupted CFile in filesystem block: 0000000000000224 I0720 00:44:27.711469 6547 ts_tablet_manager.cc:1475] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: failing tablet I0720 00:44:27.711724 7292 tablet_replica.cc:273] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275: stopping tablet replica W0720 00:44:27.711652 6547 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "bf4081672af8427b9d5d11d8dfeae031" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437882495889408 propagated_timestamp: 6404437883732606976 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\222\312\311\351\005\"3\n\005slave\022*\n 5577622f8e4e4037809dcc8abdd63b11\020\001\030\001 \001(\001" signature: "P\207\272\177\257\232\261\000U.\334`\210\2562k\245\026\367vm\001\355rwt)4\260\207\026\3517\236\206\307\324wS\320\037o\010o\201)**v\302\035\0039\342\270\373\r\030\317\rhc\006s" signing_key_seq_num: 0 } } call_seq_id: 0: Corruption: checksum error on CFile block 0000000000000224 at offset=1800 size=24: Checksum does not match: 1861669833 vs expected 1861669833 I0720 00:44:27.711855 7292 raft_consensus.cc:2147] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:27.712142 7292 raft_consensus.cc:2174] T bf4081672af8427b9d5d11d8dfeae031 P b88011e9e0d24f79882735c95c8fc275 [term 1 FOLLOWER]: Raft consensus is shut down! W0720 00:44:27.751224 7266 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:27.886310 6648 consensus_queue.cc:544] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:27.886430 6648 consensus_peers.cc:458] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:27.888248 6465 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 7139dc2492b147989250601c720f324c (attempt 1) I0720 00:44:27.889245 6717 consensus_queue.cc:206] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } W0720 00:44:27.889914 7279 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:27.890961 6839 raft_consensus.cc:1184] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch) I0720 00:44:27.891458 7266 consensus_queue.cc:984] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:27.891572 6772 consensus_queue.cc:544] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:27.891654 6772 consensus_peers.cc:458] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:27.893236 6465 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet f49376a672014e499d7fc53f9325c6f6 (attempt 1) I0720 00:44:27.893509 7293 raft_consensus.cc:2792] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.893980 6840 raft_consensus.cc:2792] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.894269 6839 consensus_queue.cc:206] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:27.897301 6717 raft_consensus.cc:1184] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch) I0720 00:44:27.897188 6465 catalog_manager.cc:4067] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.897435 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 7139dc2492b147989250601c720f324c with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:27.897791 7279 consensus_queue.cc:984] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:27.899669 7295 raft_consensus.cc:2792] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.900189 6717 raft_consensus.cc:2792] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:27.902386 6451 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet f49376a672014e499d7fc53f9325c6f6 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:27.904342 6470 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7139dc2492b147989250601c720f324c (attempt 1) I0720 00:44:27.905331 6717 consensus_queue.cc:206] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.108, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:27.905347 6470 catalog_manager.cc:4067] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index -1 to 108, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:27.906833 6839 raft_consensus.cc:1184] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 109. (index mismatch) W0720 00:44:27.907037 6651 consensus_peers.cc:458] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7139dc2492b147989250601c720f324c. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:27.907305 7266 consensus_queue.cc:984] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s I0720 00:44:27.907735 6470 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet f49376a672014e499d7fc53f9325c6f6 (attempt 1) I0720 00:44:27.908887 6840 consensus_queue.cc:206] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 1.108, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:27.908877 7266 raft_consensus.cc:2792] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 [term 1 LEADER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:27.909459 6839 raft_consensus.cc:2792] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:27.910400 6717 raft_consensus.cc:1184] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Refusing update from remote peer 5aa5a77f1d2c4510a9a7e49bc2bb32aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 108. Preceding OpId from leader: term: 1 index: 109. (index mismatch) W0720 00:44:27.910423 6775 consensus_peers.cc:458] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f49376a672014e499d7fc53f9325c6f6. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:27.910893 7295 consensus_queue.cc:984] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s I0720 00:44:27.911614 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7139dc2492b147989250601c720f324c with cas_config_opid_index 108: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:27.912734 7295 raft_consensus.cc:2792] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 LEADER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:27.913352 6465 catalog_manager.cc:4067] T 7139dc2492b147989250601c720f324c P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:27.915071 6717 raft_consensus.cc:2792] T f49376a672014e499d7fc53f9325c6f6 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Committing config change with OpId 1.109: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:27.915650 6451 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f49376a672014e499d7fc53f9325c6f6 with cas_config_opid_index 108: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:27.917626 6470 catalog_manager.cc:4067] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa reported cstate change: config changed from index 108 to 109, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0720 00:44:27.930157 7302 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.041018 7302 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.063928 7295 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.117676 7302 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.150854 7302 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.182633 7302 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.397740 7295 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.426818 7295 consensus_peers.cc:251] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.437597 7302 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:28.463732 234 disk_failure-itest.cc:256] Currently has 10 failed tablets I0720 00:44:28.463784 234 disk_failure-itest.cc:234] Resetting error injection flags W0720 00:44:28.464566 6632 generic_service.cc:128] {username='slave'} at 127.0.0.1:48978 forcing change of non-runtime-safe flag env_inject_eio I0720 00:44:28.464640 6632 generic_service.cc:148] {username='slave'} at 127.0.0.1:48978 changed flags via RPC: env_inject_eio from '0' to '0.0' W0720 00:44:28.465659 6632 generic_service.cc:128] {username='slave'} at 127.0.0.1:48978 forcing change of non-runtime-safe flag cfile_inject_corruption I0720 00:44:28.465718 6632 generic_service.cc:148] {username='slave'} at 127.0.0.1:48978 changed flags via RPC: cfile_inject_corruption from '1' to '0.0' W0720 00:44:28.466575 6632 generic_service.cc:128] {username='slave'} at 127.0.0.1:48978 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:28.466634 6632 generic_service.cc:148] {username='slave'} at 127.0.0.1:48978 changed flags via RPC: enable_tablet_copy from 'false' to 'true' W0720 00:44:28.474910 7302 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.512477 7302 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.521139 7302 consensus_peers.cc:251] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.558053 7295 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.695291 7302 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.800498 7302 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.880798 7295 consensus_peers.cc:251] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.896235 7295 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:28.953820 7302 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:28.960412 7295 raft_consensus.cc:472] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:28.960527 7295 raft_consensus.cc:494] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:28.961019 7295 leader_election.cc:296] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329) I0720 00:44:28.961455 6717 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "29cb3a99090f4ceeb99efc07786b6426" is_pre_election: true I0720 00:44:28.961501 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:28.961640 6717 raft_consensus.cc:2311] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5aa5a77f1d2c4510a9a7e49bc2bb32aa for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:28.962023 6772 leader_election.cc:348] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:28.962126 6772 leader_election.cc:310] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: 29cb3a99090f4ceeb99efc07786b6426, b88011e9e0d24f79882735c95c8fc275 I0720 00:44:28.962368 7295 raft_consensus.cc:2592] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:29.033298 7302 consensus_peers.cc:251] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.085911 7302 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.101274 7302 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.107115 7302 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.110729 7295 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.201092 7302 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:29.210227 7302 raft_consensus.cc:472] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:29.210382 7302 raft_consensus.cc:494] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:29.210913 7302 leader_election.cc:296] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:29.211527 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" is_pre_election: true I0720 00:44:29.211551 6839 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" is_pre_election: true I0720 00:44:29.211735 6839 raft_consensus.cc:2350] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 1. W0720 00:44:29.212018 6648 leader_election.cc:348] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:29.212204 6648 leader_election.cc:310] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:29.212447 7302 raft_consensus.cc:2644] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:29.212497 7302 raft_consensus.cc:472] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b88011e9e0d24f79882735c95c8fc275) I0720 00:44:29.212543 7302 raft_consensus.cc:2890] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:29.214032 7302 raft_consensus.cc:494] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:29.214474 7302 leader_election.cc:296] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Requested vote from peers b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519), 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005) I0720 00:44:29.214893 6589 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "b88011e9e0d24f79882735c95c8fc275" I0720 00:44:29.214947 6839 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bf4081672af8427b9d5d11d8dfeae031" candidate_uuid: "29cb3a99090f4ceeb99efc07786b6426" candidate_term: 2 candidate_status { last_received { term: 1 index: 107 } } ignore_live_leader: false dest_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" I0720 00:44:29.215098 6839 raft_consensus.cc:2890] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:29.215258 6648 leader_election.cc:348] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Illegal state: cannot vote while shut down I0720 00:44:29.216534 6839 raft_consensus.cc:2350] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29cb3a99090f4ceeb99efc07786b6426 in term 2. I0720 00:44:29.216850 6648 leader_election.cc:310] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29cb3a99090f4ceeb99efc07786b6426, 5aa5a77f1d2c4510a9a7e49bc2bb32aa; no voters: b88011e9e0d24f79882735c95c8fc275 I0720 00:44:29.217065 7302 raft_consensus.cc:2644] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:29.217355 7302 raft_consensus.cc:667] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Becoming Leader. State: Replica: 29cb3a99090f4ceeb99efc07786b6426, State: Running, Role: LEADER I0720 00:44:29.217530 7302 consensus_queue.cc:206] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:29.219615 6470 catalog_manager.cc:4067] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: term changed from 1 to 2, leader changed from b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) to 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130). New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b88011e9e0d24f79882735c95c8fc275" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 41519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: UNKNOWN } } } W0720 00:44:29.279422 7295 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 62dd01b770a74740ac39ccdea33401d8 | 127.0.58.190:37529 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 29cb3a99090f4ceeb99efc07786b6426 | 127.0.58.130:42329 | HEALTHY | 5aa5a77f1d2c4510a9a7e49bc2bb32aa | 127.0.58.131:38005 | HEALTHY | b88011e9e0d24f79882735c95c8fc275 | 127.0.58.129:41519 | HEALTHY | e1d7982f305c4a728a23a075e1e9ac7b | 127.0.58.132:33507 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:41519 enable_tablet_copy | false | unsafe | 127.0.58.130:42329, 127.0.58.131:38005, 127.0.58.132:33507 enable_tablet_copy | true | unsafe | 127.0.58.129:41519 env_inject_eio | 0 | hidden | 127.0.58.129:41519 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:41519 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:42329 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:38005 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:33507 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:41519 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:42329 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:38005 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:33507 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet bf4081672af8427b9d5d11d8dfeae031 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: failing tablet 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329): RUNNING [LEADER] 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005): RUNNING All reported replicas are: A = b88011e9e0d24f79882735c95c8fc275 B = 29cb3a99090f4ceeb99efc07786b6426 C = 5aa5a77f1d2c4510a9a7e49bc2bb32aa The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 2 | -1 | Yes C | A B C | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 9 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 9 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | b88011e9e0d24f79882735c95c8fc275 | 127.0.58.129:41519 | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:29.311395 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors W0720 00:44:29.476056 7302 consensus_peers.cc:251] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.502816 7295 consensus_peers.cc:251] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 62dd01b770a74740ac39ccdea33401d8 | 127.0.58.190:37529 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 29cb3a99090f4ceeb99efc07786b6426 | 127.0.58.130:42329 | HEALTHY | 5aa5a77f1d2c4510a9a7e49bc2bb32aa | 127.0.58.131:38005 | HEALTHY | b88011e9e0d24f79882735c95c8fc275 | 127.0.58.129:41519 | HEALTHY | e1d7982f305c4a728a23a075e1e9ac7b | 127.0.58.132:33507 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:41519 enable_tablet_copy | false | unsafe | 127.0.58.130:42329, 127.0.58.131:38005, 127.0.58.132:33507 enable_tablet_copy | true | unsafe | 127.0.58.129:41519 env_inject_eio | 0 | hidden | 127.0.58.129:41519 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:41519 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:42329 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:38005 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:33507 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:41519 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:42329 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:38005 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:33507 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet bf4081672af8427b9d5d11d8dfeae031 of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: failing tablet 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329): RUNNING [LEADER] 5aa5a77f1d2c4510a9a7e49bc2bb32aa (127.0.58.131:38005): RUNNING All reported replicas are: A = b88011e9e0d24f79882735c95c8fc275 B = 29cb3a99090f4ceeb99efc07786b6426 C = 5aa5a77f1d2c4510a9a7e49bc2bb32aa The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B* C | | | Yes A | [config not available] | | | B | A B* C | 2 | -1 | Yes C | A B C | 2 | -1 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | UNDER_REPLICATED | 10 | 9 | 0 | 1 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 9 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | b88011e9e0d24f79882735c95c8fc275 | 127.0.58.129:41519 | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED W0720 00:44:29.530609 7302 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:29.530660 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0720 00:44:29.596091 6839 raft_consensus.cc:1184] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) I0720 00:44:29.596591 7302 consensus_queue.cc:984] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:29.601053 6648 consensus_queue.cc:544] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: The tablet replica hosted on peer b88011e9e0d24f79882735c95c8fc275 has failed W0720 00:44:29.601147 6648 consensus_peers.cc:458] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer b88011e9e0d24f79882735c95c8fc275 (127.0.58.129:41519): Couldn't send request to peer b88011e9e0d24f79882735c95c8fc275. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:29.602744 6465 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet bf4081672af8427b9d5d11d8dfeae031 (attempt 1) I0720 00:44:29.603641 6717 consensus_queue.cc:206] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 108, Committed index: 108, Last appended: 2.108, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } I0720 00:44:29.604763 6839 raft_consensus.cc:1184] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 2 index: 108. Preceding OpId from leader: term: 2 index: 109. (index mismatch) I0720 00:44:29.605134 7302 consensus_queue.cc:984] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 109, Last known committed idx: 108, Time since last communication: 0.000s I0720 00:44:29.606673 7304 raft_consensus.cc:2792] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Committing config change with OpId 2.109: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } I0720 00:44:29.607163 6839 raft_consensus.cc:2792] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Committing config change with OpId 2.109: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New config: { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } } W0720 00:44:29.607460 7302 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:29.609472 6453 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet bf4081672af8427b9d5d11d8dfeae031 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) W0720 00:44:29.610316 7302 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:29.610442 6465 catalog_manager.cc:4067] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index -1 to 109, VOTER b88011e9e0d24f79882735c95c8fc275 (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 109 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } } I0720 00:44:29.612772 6465 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet bf4081672af8427b9d5d11d8dfeae031 (attempt 1) I0720 00:44:29.613828 6717 consensus_queue.cc:206] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 2.109, Last appended by leader: 107, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } I0720 00:44:29.615221 6839 raft_consensus.cc:1184] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Refusing update from remote peer 29cb3a99090f4ceeb99efc07786b6426: Log matching property violated. Preceding OpId in replica: term: 2 index: 109. Preceding OpId from leader: term: 2 index: 110. (index mismatch) I0720 00:44:29.615588 7304 consensus_queue.cc:984] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s W0720 00:44:29.615746 6651 consensus_peers.cc:458] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Couldn't send request to peer e1d7982f305c4a728a23a075e1e9ac7b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf4081672af8427b9d5d11d8dfeae031. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:29.617079 7302 raft_consensus.cc:2792] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 [term 2 LEADER]: Committing config change with OpId 2.110: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:29.617574 6839 raft_consensus.cc:2792] T bf4081672af8427b9d5d11d8dfeae031 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa [term 2 FOLLOWER]: Committing config change with OpId 2.110: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } } } I0720 00:44:29.619705 6453 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bf4081672af8427b9d5d11d8dfeae031 with cas_config_opid_index 109: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:29.620921 6465 catalog_manager.cc:4067] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 reported cstate change: config changed from index 109 to 110, NON_VOTER e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "29cb3a99090f4ceeb99efc07786b6426" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "29cb3a99090f4ceeb99efc07786b6426" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa5a77f1d2c4510a9a7e49bc2bb32aa" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d7982f305c4a728a23a075e1e9ac7b" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 33507 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0720 00:44:29.711913 7304 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.715611 7295 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.722043 7295 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:29.732015 7304 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 62dd01b770a74740ac39ccdea33401d8 | 127.0.58.190:37529 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 29cb3a99090f4ceeb99efc07786b6426 | 127.0.58.130:42329 | HEALTHY | 5aa5a77f1d2c4510a9a7e49bc2bb32aa | 127.0.58.131:38005 | HEALTHY | b88011e9e0d24f79882735c95c8fc275 | 127.0.58.129:41519 | HEALTHY | e1d7982f305c4a728a23a075e1e9ac7b | 127.0.58.132:33507 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:41519 enable_tablet_copy | false | unsafe | 127.0.58.130:42329, 127.0.58.131:38005, 127.0.58.132:33507 enable_tablet_copy | true | unsafe | 127.0.58.129:41519 env_inject_eio | 0 | hidden | 127.0.58.129:41519 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:41519 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:42329 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:38005 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:33507 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:41519 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:42329 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:38005 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_0.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:33507 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 10 | 10 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | b88011e9e0d24f79882735c95c8fc275 | 127.0.58.129:41519 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:44:29.824044 234 log_verifier.cc:121] Checking tablet 27478acfc8ee4027a84efa2c19ee11e9 I0720 00:44:29.879982 234 log_verifier.cc:172] Verified matching terms for 109 ops in tablet 27478acfc8ee4027a84efa2c19ee11e9 I0720 00:44:29.880167 234 log_verifier.cc:121] Checking tablet 49ad7f72eaa04a14bc64fa4f6cbce73c W0720 00:44:29.939608 7295 consensus_peers.cc:251] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:29.947028 234 log_verifier.cc:172] Verified matching terms for 109 ops in tablet 49ad7f72eaa04a14bc64fa4f6cbce73c I0720 00:44:29.947157 234 log_verifier.cc:121] Checking tablet 52408e87c8e54a7394a12dc191a9b4a4 W0720 00:44:29.999781 7304 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.004462 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet 52408e87c8e54a7394a12dc191a9b4a4 I0720 00:44:30.004596 234 log_verifier.cc:121] Checking tablet 57d8956326d345eba0ced816663169f9 W0720 00:44:30.028846 7304 consensus_peers.cc:251] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.045166 7304 consensus_peers.cc:251] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.054455 7304 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.062346 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 57d8956326d345eba0ced816663169f9 I0720 00:44:30.062469 234 log_verifier.cc:121] Checking tablet 7139dc2492b147989250601c720f324c I0720 00:44:30.122197 234 log_verifier.cc:172] Verified matching terms for 109 ops in tablet 7139dc2492b147989250601c720f324c I0720 00:44:30.122371 234 log_verifier.cc:121] Checking tablet 79f36e06977e420f94ab39786832ecc9 W0720 00:44:30.174801 7295 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.182634 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet 79f36e06977e420f94ab39786832ecc9 I0720 00:44:30.182796 234 log_verifier.cc:121] Checking tablet a5154a3c09a5435fb2ac44744e8e7007 W0720 00:44:30.195039 7295 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.197161 7304 consensus_peers.cc:251] T 57d8956326d345eba0ced816663169f9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.242811 234 log_verifier.cc:172] Verified matching terms for 110 ops in tablet a5154a3c09a5435fb2ac44744e8e7007 I0720 00:44:30.242974 234 log_verifier.cc:121] Checking tablet bf4081672af8427b9d5d11d8dfeae031 W0720 00:44:30.275354 7304 consensus_peers.cc:251] T 49ad7f72eaa04a14bc64fa4f6cbce73c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.280771 7304 consensus_peers.cc:251] T 52408e87c8e54a7394a12dc191a9b4a4 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.302834 234 log_verifier.cc:172] Verified matching terms for 110 ops in tablet bf4081672af8427b9d5d11d8dfeae031 I0720 00:44:30.302989 234 log_verifier.cc:121] Checking tablet d4520f8364cb4356bf859651b4a349b2 W0720 00:44:30.351783 7295 consensus_peers.cc:251] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.374979 234 log_verifier.cc:172] Verified matching terms for 110 ops in tablet d4520f8364cb4356bf859651b4a349b2 I0720 00:44:30.375129 234 log_verifier.cc:121] Checking tablet f49376a672014e499d7fc53f9325c6f6 W0720 00:44:30.429790 7304 consensus_peers.cc:251] T 27478acfc8ee4027a84efa2c19ee11e9 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.432904 234 log_verifier.cc:172] Verified matching terms for 109 ops in tablet f49376a672014e499d7fc53f9325c6f6 I0720 00:44:30.433622 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 6504 I0720 00:44:30.434568 6632 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:30.453088 7304 consensus_peers.cc:251] T d4520f8364cb4356bf859651b4a349b2 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.531595 7304 consensus_peers.cc:251] T bf4081672af8427b9d5d11d8dfeae031 P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.588136 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 6504 W0720 00:44:30.612687 6454 connection.cc:655] server connection from 127.0.58.129:52513 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:44:30.612916 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 6639 I0720 00:44:30.613646 6757 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:30.656489 7295 consensus_peers.cc:251] T a5154a3c09a5435fb2ac44744e8e7007 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.765792 7304 consensus_peers.cc:251] T 7139dc2492b147989250601c720f324c P 29cb3a99090f4ceeb99efc07786b6426 -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.766114 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 6639 W0720 00:44:30.771629 7295 consensus_peers.cc:251] T f49376a672014e499d7fc53f9325c6f6 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:30.789729 6774 connection.cc:655] client connection to 127.0.58.130:42329 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0720 00:44:30.789916 6774 consensus_peers.cc:458] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer 29cb3a99090f4ceeb99efc07786b6426 (127.0.58.130:42329): Couldn't send request to peer 29cb3a99090f4ceeb99efc07786b6426. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W0720 00:44:30.789949 6451 connection.cc:655] server connection from 127.0.58.130:58213 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:44:30.790170 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 6763 I0720 00:44:30.790953 6881 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:30.943274 7295 consensus_peers.cc:251] T 79f36e06977e420f94ab39786832ecc9 P 5aa5a77f1d2c4510a9a7e49bc2bb32aa -> Peer e1d7982f305c4a728a23a075e1e9ac7b (127.0.58.132:33507): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:30.944087 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 6763 I0720 00:44:30.966503 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 7106 I0720 00:44:30.967396 7225 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:31.009987 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 7106 I0720 00:44:31.020171 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 6442 I0720 00:44:31.021046 6500 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:31.104028 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 6442 [ OK ] DiskError/DiskErrorITest.TestFailDuringScanWorkload/0 (15743 ms) [ RUN ] DiskError/DiskErrorITest.TestFailDuringScanWorkload/1 I0720 00:44:31.148491 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-2 --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:34075 --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --master_tombstone_evicted_tablet_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:31.342511 7398 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:31.342712 7398 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:31.345211 7398 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0720 00:44:31.345252 7398 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:31.345280 7398 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0720 00:44:31.345310 7398 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0720 00:44:31.349709 7398 master_main.cc:98] Master server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/wal --master_tombstone_evicted_tablet_replicas=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.0.58.190:34075 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb --webserver_interface=127.0.58.190 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:31.356214 7398 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 97016us I0720 00:44:31.357388 7398 fs_manager.cc:263] Metadata directory not provided I0720 00:44:31.357460 7398 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:31.357772 7398 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:31.357844 7398 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:31.360237 7398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/instance: uuid: "4145b8157484456995b94b60104ebd7e" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.360776 7398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-1/instance: uuid: "4145b8157484456995b94b60104ebd7e" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.361129 7398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-2/instance: uuid: "4145b8157484456995b94b60104ebd7e" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.361452 7398 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/wal/instance: uuid: "4145b8157484456995b94b60104ebd7e" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.370158 7398 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s I0720 00:44:31.371392 7398 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:31.371913 7398 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:31.375059 7398 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:44:31.375121 7398 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/wal uuid: "4145b8157484456995b94b60104ebd7e" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.375281 7398 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:31.414834 7398 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:31.427964 7398 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.190:34075 I0720 00:44:31.429096 7398 webserver.cc:374] Webserver started at http://127.0.58.190:40793/ using document root and password file I0720 00:44:31.429800 7398 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb I0720 00:44:31.431726 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master as pid 7398 I0720 00:44:31.432354 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.129:0 --local_ip_for_outbound_sockets=127.0.58.129 --webserver_interface=127.0.58.129 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34075 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:31.443238 7459 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e: Bootstrap starting. I0720 00:44:31.445731 7459 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e: No blocks or log segments found. Creating new log. I0720 00:44:31.446633 7459 log.cc:525] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e: Log is configured to *not* fsync() on all Append() calls I0720 00:44:31.448078 7459 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e: No bootstrap required, opened a new log I0720 00:44:31.454435 7459 raft_consensus.cc:340] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } I0720 00:44:31.454676 7459 raft_consensus.cc:366] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:31.454756 7459 raft_consensus.cc:705] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4145b8157484456995b94b60104ebd7e, State: Initialized, Role: FOLLOWER I0720 00:44:31.455271 7459 consensus_queue.cc:229] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } I0720 00:44:31.455456 7459 raft_consensus.cc:380] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0720 00:44:31.455523 7459 raft_consensus.cc:472] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0720 00:44:31.455646 7459 raft_consensus.cc:2890] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:31.457206 7459 raft_consensus.cc:494] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } I0720 00:44:31.457633 7459 leader_election.cc:310] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4145b8157484456995b94b60104ebd7e; no voters: I0720 00:44:31.458364 7462 raft_consensus.cc:2644] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:31.458688 7459 leader_election.cc:296] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [CANDIDATE]: Term 1 election: Requested vote from peers I0720 00:44:31.459444 7462 raft_consensus.cc:667] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [term 1 LEADER]: Becoming Leader. State: Replica: 4145b8157484456995b94b60104ebd7e, State: Running, Role: LEADER I0720 00:44:31.459792 7462 consensus_queue.cc:206] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } I0720 00:44:31.462926 7463 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4145b8157484456995b94b60104ebd7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } } I0720 00:44:31.463443 7463 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: This master's current role is: LEADER I0720 00:44:31.463228 7464 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4145b8157484456995b94b60104ebd7e. Latest consensus state: current_term: 1 leader_uuid: "4145b8157484456995b94b60104ebd7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } } I0720 00:44:31.464567 7464 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: This master's current role is: LEADER I0720 00:44:31.464596 7459 sys_catalog.cc:340] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "4145b8157484456995b94b60104ebd7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4145b8157484456995b94b60104ebd7e" member_type: VOTER } } I0720 00:44:31.464794 7459 sys_catalog.cc:343] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: This master's current role is: LEADER I0720 00:44:31.466666 7459 sys_catalog.cc:428] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e [sys.catalog]: configured and running, proceeding with master startup. I0720 00:44:31.467160 7468 catalog_manager.cc:1064] Loading table and tablet metadata into memory... I0720 00:44:31.476104 7468 catalog_manager.cc:1075] Initializing Kudu internal certificate authority... I0720 00:44:31.539742 7468 catalog_manager.cc:959] Generated new certificate authority record I0720 00:44:31.541330 7468 catalog_manager.cc:1084] Loading token signing keys... I0720 00:44:31.553472 7468 catalog_manager.cc:4371] T 00000000000000000000000000000000 P 4145b8157484456995b94b60104ebd7e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:31.601644 7460 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:31.601874 7460 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:31.604027 7460 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:31.604122 7460 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:31.604213 7460 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:31.604291 7460 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.129 I0720 00:44:31.610893 7460 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb --webserver_interface=127.0.58.129 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:34075 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.129 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:31.618763 7460 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 97016us I0720 00:44:31.619437 7460 fs_manager.cc:263] Metadata directory not provided I0720 00:44:31.619515 7460 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:31.619801 7460 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:31.619877 7460 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:31.622040 7460 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/instance: uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.622609 7460 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/instance: uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.622952 7460 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-2/instance: uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.623276 7460 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/wal/instance: uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.635329 7460 fs_manager.cc:503] Time spent creating directory manager: real 0.012s user 0.005s sys 0.007s I0720 00:44:31.636611 7460 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:31.637138 7460 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:31.643890 7460 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s I0720 00:44:31.643959 7460 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/wal uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" format_stamp: "Formatted at 2019-07-20 00:44:31 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:31.644134 7460 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:31.674558 7460 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:31.679093 7460 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:31.679158 7460 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:31.864285 7460 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.129:42263 I0720 00:44:31.865285 7460 webserver.cc:374] Webserver started at http://127.0.58.129:46437/ using document root and password file I0720 00:44:31.865905 7460 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb I0720 00:44:31.868065 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 7460 I0720 00:44:31.868768 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.130:0 --local_ip_for_outbound_sockets=127.0.58.130 --webserver_interface=127.0.58.130 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34075 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:31.883190 7592 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34075 I0720 00:44:31.883394 7592 heartbeater.cc:405] Registering TS with master... I0720 00:44:31.883929 7592 heartbeater.cc:454] Master 127.0.58.190:34075 requested a full tablet report, sending... I0720 00:44:31.885555 7426 ts_manager.cc:161] Registered new tserver with Master: 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:31.886968 7426 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.129:54125 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:32.030936 7596 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:32.031167 7596 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:32.033222 7596 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:32.033313 7596 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:32.033365 7596 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:32.033439 7596 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.130 I0720 00:44:32.037184 7596 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb --webserver_interface=127.0.58.130 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:34075 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.130 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:32.044737 7596 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 97516us I0720 00:44:32.045373 7596 fs_manager.cc:263] Metadata directory not provided I0720 00:44:32.045435 7596 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:32.045706 7596 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:32.045768 7596 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:32.048015 7596 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/instance: uuid: "7a005754141d43ac91f33e8797a489ab" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.048527 7596 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-1/instance: uuid: "7a005754141d43ac91f33e8797a489ab" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.048836 7596 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-2/instance: uuid: "7a005754141d43ac91f33e8797a489ab" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.049119 7596 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/wal/instance: uuid: "7a005754141d43ac91f33e8797a489ab" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.057720 7596 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s I0720 00:44:32.058979 7596 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:32.059439 7596 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:32.062225 7596 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0720 00:44:32.062281 7596 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/wal uuid: "7a005754141d43ac91f33e8797a489ab" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.062422 7596 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:32.101382 7596 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:32.102913 7596 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:32.102965 7596 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:32.131793 7596 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.130:40487 I0720 00:44:32.132902 7596 webserver.cc:374] Webserver started at http://127.0.58.130:38033/ using document root and password file I0720 00:44:32.133559 7596 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb I0720 00:44:32.135076 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 7596 I0720 00:44:32.135718 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.131:0 --local_ip_for_outbound_sockets=127.0.58.131 --webserver_interface=127.0.58.131 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34075 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:32.150816 7717 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34075 I0720 00:44:32.151028 7717 heartbeater.cc:405] Registering TS with master... I0720 00:44:32.151589 7717 heartbeater.cc:454] Master 127.0.58.190:34075 requested a full tablet report, sending... I0720 00:44:32.152707 7426 ts_manager.cc:161] Registered new tserver with Master: 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.153527 7426 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.130:59015 WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:32.299116 7720 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:32.299340 7720 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0720 00:44:32.301465 7720 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:32.301554 7720 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:32.301604 7720 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:32.301679 7720 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.131 I0720 00:44:32.305694 7720 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb --webserver_interface=127.0.58.131 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:34075 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.131 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:32.313200 7720 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 97516us I0720 00:44:32.313876 7720 fs_manager.cc:263] Metadata directory not provided I0720 00:44:32.313946 7720 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:32.314318 7720 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:32.314389 7720 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:32.316627 7720 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/instance: uuid: "c3780073b3594c88b678ec4da2bd8cda" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.317973 7720 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-1/instance: uuid: "c3780073b3594c88b678ec4da2bd8cda" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.318428 7720 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-2/instance: uuid: "c3780073b3594c88b678ec4da2bd8cda" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.318807 7720 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/wal/instance: uuid: "c3780073b3594c88b678ec4da2bd8cda" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.327319 7720 fs_manager.cc:503] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s I0720 00:44:32.328840 7720 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:32.329340 7720 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:32.332412 7720 fs_manager.cc:419] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0720 00:44:32.332471 7720 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/wal uuid: "c3780073b3594c88b678ec4da2bd8cda" format_stamp: "Formatted at 2019-07-20 00:44:32 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:32.332635 7720 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:32.358952 7720 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:32.360438 7720 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:32.360496 7720 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:32.383395 7720 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.131:37319 I0720 00:44:32.384393 7720 webserver.cc:374] Webserver started at http://127.0.58.131:45211/ using document root and password file I0720 00:44:32.384999 7720 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb I0720 00:44:32.393359 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 7720 I0720 00:44:32.400053 7841 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34075 I0720 00:44:32.400246 7841 heartbeater.cc:405] Registering TS with master... I0720 00:44:32.400786 7841 heartbeater.cc:454] Master 127.0.58.190:34075 requested a full tablet report, sending... I0720 00:44:32.401938 7426 ts_manager.cc:161] Registered new tserver with Master: c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.402782 7426 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.131:50465 I0720 00:44:32.405504 234 external_mini_cluster.cc:628] 3 TS(s) registered with all masters I0720 00:44:32.414433 234 test_util.cc:212] Using random seed: 628305620 I0720 00:44:32.434501 7426 catalog_manager.cc:1418] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47124: name: "test-workload" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s" } partition_schema { range_schema { columns { name: "key" } } } W0720 00:44:32.436117 7426 catalog_manager.cc:1583] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0720 00:44:32.469262 7654 tablet_service.cc:1022] Processing CreateTablet for tablet 0d5f343fde1d404598e962001ef17ae2 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:32.470645 7771 tablet_service.cc:1022] Processing CreateTablet for tablet c8db08710314422cbd1b616d57ec4cec (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:32.473057 7777 tablet_service.cc:1022] Processing CreateTablet for tablet a487d770322e4226b00934877a3bff5d (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:32.473156 7651 tablet_service.cc:1022] Processing CreateTablet for tablet bd37441ffc54486b90ba5230a81da533 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:32.473177 7520 tablet_service.cc:1022] Processing CreateTablet for tablet 7b95ae04955844828ac3ced5e85c9ed5 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:32.473762 7524 tablet_service.cc:1022] Processing CreateTablet for tablet 9a5c9555b3fa4c92940250278686a179 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:32.473965 7772 tablet_service.cc:1022] Processing CreateTablet for tablet eb86205d3d8548f4b0daaa602f89c3a9 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:32.473441 7521 tablet_service.cc:1022] Processing CreateTablet for tablet 0380f4644f22404e9f2b80565c2d7a0b (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:32.474195 7650 tablet_service.cc:1022] Processing CreateTablet for tablet 4d36dab0fb81459dbc2df72f66852ea2 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:32.473518 7526 tablet_service.cc:1022] Processing CreateTablet for tablet bd37441ffc54486b90ba5230a81da533 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:32.473369 7528 tablet_service.cc:1022] Processing CreateTablet for tablet a487d770322e4226b00934877a3bff5d (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:32.472837 7527 tablet_service.cc:1022] Processing CreateTablet for tablet 5570d23334524c9a9e1bbbdf1c4c3806 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:32.474009 7652 tablet_service.cc:1022] Processing CreateTablet for tablet 5570d23334524c9a9e1bbbdf1c4c3806 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:32.474834 7776 tablet_service.cc:1022] Processing CreateTablet for tablet 5570d23334524c9a9e1bbbdf1c4c3806 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092 I0720 00:44:32.469980 7653 tablet_service.cc:1022] Processing CreateTablet for tablet a487d770322e4226b00934877a3bff5d (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728 I0720 00:44:32.475446 7778 tablet_service.cc:1022] Processing CreateTablet for tablet 0d5f343fde1d404598e962001ef17ae2 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:32.476080 7773 tablet_service.cc:1022] Processing CreateTablet for tablet 9a5c9555b3fa4c92940250278686a179 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:32.474579 7522 tablet_service.cc:1022] Processing CreateTablet for tablet c8db08710314422cbd1b616d57ec4cec (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:32.476080 7523 tablet_service.cc:1022] Processing CreateTablet for tablet eb86205d3d8548f4b0daaa602f89c3a9 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:32.476909 7774 tablet_service.cc:1022] Processing CreateTablet for tablet 4d36dab0fb81459dbc2df72f66852ea2 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:32.476985 7529 tablet_service.cc:1022] Processing CreateTablet for tablet 0d5f343fde1d404598e962001ef17ae2 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION VALUES < 214748364 I0720 00:44:32.477649 7775 tablet_service.cc:1022] Processing CreateTablet for tablet bd37441ffc54486b90ba5230a81da533 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456 I0720 00:44:32.483151 7525 tablet_service.cc:1022] Processing CreateTablet for tablet 4d36dab0fb81459dbc2df72f66852ea2 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820 I0720 00:44:32.488860 7649 tablet_service.cc:1022] Processing CreateTablet for tablet 9a5c9555b3fa4c92940250278686a179 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184 I0720 00:44:32.490579 7648 tablet_service.cc:1022] Processing CreateTablet for tablet eb86205d3d8548f4b0daaa602f89c3a9 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548 I0720 00:44:32.492735 7647 tablet_service.cc:1022] Processing CreateTablet for tablet c8db08710314422cbd1b616d57ec4cec (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912 I0720 00:44:32.493750 7770 tablet_service.cc:1022] Processing CreateTablet for tablet 0380f4644f22404e9f2b80565c2d7a0b (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:32.494720 7769 tablet_service.cc:1022] Processing CreateTablet for tablet 7b95ae04955844828ac3ced5e85c9ed5 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:32.495664 7646 tablet_service.cc:1022] Processing CreateTablet for tablet 0380f4644f22404e9f2b80565c2d7a0b (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276 I0720 00:44:32.507849 7645 tablet_service.cc:1022] Processing CreateTablet for tablet 7b95ae04955844828ac3ced5e85c9ed5 (table=test-workload [id=db2f82abcef04875b1b1ca569ecb5d7e]), partition=RANGE (key) PARTITION VALUES >= 1932735276 I0720 00:44:32.574721 7866 tablet_bootstrap.cc:439] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.575243 7867 tablet_bootstrap.cc:439] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.579746 7867 tablet_bootstrap.cc:590] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.585814 7869 tablet_bootstrap.cc:439] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.588464 7868 tablet_bootstrap.cc:439] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.592155 7868 tablet_bootstrap.cc:590] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.593681 7867 log.cc:525] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Log is configured to *not* fsync() on all Append() calls I0720 00:44:32.592483 7870 tablet_bootstrap.cc:439] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.594327 7869 tablet_bootstrap.cc:590] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.580261 7865 tablet_bootstrap.cc:439] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.594830 7866 tablet_bootstrap.cc:590] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.597121 7868 log.cc:525] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda: Log is configured to *not* fsync() on all Append() calls I0720 00:44:32.597196 7865 tablet_bootstrap.cc:590] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.597858 7871 tablet_bootstrap.cc:439] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.598037 7872 tablet_bootstrap.cc:439] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.598068 7870 tablet_bootstrap.cc:590] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.598702 7873 tablet_bootstrap.cc:439] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.599853 7867 tablet_bootstrap.cc:439] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.600325 7869 tablet_bootstrap.cc:439] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.600574 7867 ts_tablet_manager.cc:1059] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s I0720 00:44:32.600942 7868 tablet_bootstrap.cc:439] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.601383 7868 ts_tablet_manager.cc:1059] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s I0720 00:44:32.601475 7872 tablet_bootstrap.cc:590] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.601661 7866 tablet_bootstrap.cc:439] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.601851 7865 tablet_bootstrap.cc:439] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.601959 7865 ts_tablet_manager.cc:1059] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s I0720 00:44:32.602200 7869 ts_tablet_manager.cc:1059] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s I0720 00:44:32.602347 7870 tablet_bootstrap.cc:439] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.603020 7866 ts_tablet_manager.cc:1059] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s I0720 00:44:32.603685 7871 tablet_bootstrap.cc:590] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.603755 7873 tablet_bootstrap.cc:590] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.604606 7870 ts_tablet_manager.cc:1059] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s I0720 00:44:32.604961 7871 log.cc:525] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab: Log is configured to *not* fsync() on all Append() calls I0720 00:44:32.607775 7873 tablet_bootstrap.cc:439] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.608546 7873 ts_tablet_manager.cc:1059] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s I0720 00:44:32.609403 7872 tablet_bootstrap.cc:439] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.610266 7872 ts_tablet_manager.cc:1059] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s I0720 00:44:32.612555 7866 raft_consensus.cc:340] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.613250 7866 raft_consensus.cc:366] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.613150 7867 raft_consensus.cc:340] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.613382 7866 raft_consensus.cc:705] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.613526 7867 raft_consensus.cc:366] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.613624 7867 raft_consensus.cc:705] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.612880 7865 raft_consensus.cc:340] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.614188 7865 raft_consensus.cc:366] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.614122 7866 consensus_queue.cc:229] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.614624 7867 consensus_queue.cc:229] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.614854 7865 raft_consensus.cc:705] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.615090 7871 tablet_bootstrap.cc:439] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.615267 7871 ts_tablet_manager.cc:1059] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s I0720 00:44:32.615185 7865 consensus_queue.cc:229] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.616216 7592 heartbeater.cc:446] Master 127.0.58.190:34075 was elected leader, sending a full tablet report... I0720 00:44:32.622236 7870 raft_consensus.cc:340] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.622200 7872 raft_consensus.cc:340] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.622200 7871 raft_consensus.cc:340] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.622673 7871 raft_consensus.cc:366] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.622710 7870 raft_consensus.cc:366] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.622783 7871 raft_consensus.cc:705] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.622823 7870 raft_consensus.cc:705] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.623224 7872 raft_consensus.cc:366] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.623260 7871 consensus_queue.cc:229] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.623349 7870 consensus_queue.cc:229] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.624099 7717 heartbeater.cc:446] Master 127.0.58.190:34075 was elected leader, sending a full tablet report... I0720 00:44:32.621893 7868 raft_consensus.cc:340] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.624359 7868 raft_consensus.cc:366] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.624414 7868 raft_consensus.cc:705] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.624001 7869 raft_consensus.cc:340] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.624717 7869 raft_consensus.cc:366] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.624801 7869 raft_consensus.cc:705] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.625150 7841 heartbeater.cc:446] Master 127.0.58.190:34075 was elected leader, sending a full tablet report... I0720 00:44:32.625151 7871 ts_tablet_manager.cc:1087] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.010s user 0.000s sys 0.004s I0720 00:44:32.625162 7866 ts_tablet_manager.cc:1087] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.022s user 0.000s sys 0.009s I0720 00:44:32.625464 7866 tablet_bootstrap.cc:439] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.625466 7871 tablet_bootstrap.cc:439] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.625782 7869 consensus_queue.cc:229] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.622655 7873 raft_consensus.cc:340] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.626238 7873 raft_consensus.cc:366] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.626322 7873 raft_consensus.cc:705] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.623420 7872 raft_consensus.cc:705] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.626695 7869 ts_tablet_manager.cc:1087] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s I0720 00:44:32.626881 7866 tablet_bootstrap.cc:590] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.626811 7872 consensus_queue.cc:229] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.627444 7871 tablet_bootstrap.cc:590] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.627395 7868 consensus_queue.cc:229] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.627827 7869 tablet_bootstrap.cc:439] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.627743 7873 consensus_queue.cc:229] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.627887 7883 raft_consensus.cc:472] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.628607 7868 ts_tablet_manager.cc:1087] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.027s user 0.007s sys 0.002s I0720 00:44:32.628950 7868 tablet_bootstrap.cc:439] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.630439 7868 tablet_bootstrap.cc:590] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.628113 7870 ts_tablet_manager.cc:1087] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.023s user 0.004s sys 0.004s I0720 00:44:32.631443 7870 tablet_bootstrap.cc:439] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.631814 7883 raft_consensus.cc:494] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.632231 7869 tablet_bootstrap.cc:590] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.632803 7870 tablet_bootstrap.cc:590] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.633896 7873 ts_tablet_manager.cc:1087] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s I0720 00:44:32.634233 7873 tablet_bootstrap.cc:439] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.634847 7868 tablet_bootstrap.cc:439] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log W0720 00:44:32.635265 7718 tablet.cc:1799] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:32.635429 7868 ts_tablet_manager.cc:1059] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s I0720 00:44:32.631669 7865 ts_tablet_manager.cc:1087] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.029s user 0.002s sys 0.002s I0720 00:44:32.636098 7868 raft_consensus.cc:340] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.636433 7865 tablet_bootstrap.cc:439] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.631515 7867 ts_tablet_manager.cc:1087] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.029s user 0.000s sys 0.009s I0720 00:44:32.637342 7867 tablet_bootstrap.cc:439] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. W0720 00:44:32.636636 7842 tablet.cc:1799] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:32.637423 7869 tablet_bootstrap.cc:439] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.638207 7869 ts_tablet_manager.cc:1059] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s I0720 00:44:32.638552 7872 ts_tablet_manager.cc:1087] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.017s user 0.008s sys 0.001s I0720 00:44:32.638828 7872 tablet_bootstrap.cc:439] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.638952 7869 raft_consensus.cc:340] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.639304 7869 raft_consensus.cc:366] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.639377 7869 raft_consensus.cc:705] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.639232 7868 raft_consensus.cc:366] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.639638 7868 raft_consensus.cc:705] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.639663 7873 tablet_bootstrap.cc:590] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.639880 7868 consensus_queue.cc:229] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.640214 7872 tablet_bootstrap.cc:590] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.643855 7868 ts_tablet_manager.cc:1087] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s I0720 00:44:32.644109 7868 tablet_bootstrap.cc:439] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.645658 7868 tablet_bootstrap.cc:590] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.644253 7869 consensus_queue.cc:229] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.648387 7869 ts_tablet_manager.cc:1087] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s I0720 00:44:32.648645 7869 tablet_bootstrap.cc:439] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.649430 7872 tablet_bootstrap.cc:439] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.649622 7872 ts_tablet_manager.cc:1059] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s I0720 00:44:32.649834 7869 tablet_bootstrap.cc:590] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.650727 7871 tablet_bootstrap.cc:439] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.650997 7870 tablet_bootstrap.cc:439] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.638222 7883 leader_election.cc:296] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.642627 7866 tablet_bootstrap.cc:439] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.651216 7870 ts_tablet_manager.cc:1059] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s I0720 00:44:32.651315 7866 ts_tablet_manager.cc:1059] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.001s I0720 00:44:32.651978 7870 raft_consensus.cc:340] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.652084 7866 raft_consensus.cc:340] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.652292 7870 raft_consensus.cc:366] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.652379 7866 raft_consensus.cc:366] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.652390 7870 raft_consensus.cc:705] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.652492 7866 raft_consensus.cc:705] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.652616 7867 tablet_bootstrap.cc:590] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.652616 7870 consensus_queue.cc:229] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.652814 7866 consensus_queue.cc:229] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.654515 7870 ts_tablet_manager.cc:1087] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0720 00:44:32.654799 7870 tablet_bootstrap.cc:439] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.647330 7865 tablet_bootstrap.cc:590] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.655182 7871 ts_tablet_manager.cc:1059] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.009s I0720 00:44:32.655901 7871 raft_consensus.cc:340] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.656141 7870 tablet_bootstrap.cc:590] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.656148 7871 raft_consensus.cc:366] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.656208 7871 raft_consensus.cc:705] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.656464 7866 ts_tablet_manager.cc:1087] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s I0720 00:44:32.656471 7871 consensus_queue.cc:229] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.656759 7866 tablet_bootstrap.cc:439] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.658380 7866 tablet_bootstrap.cc:590] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.658614 7871 ts_tablet_manager.cc:1087] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0720 00:44:32.658888 7871 tablet_bootstrap.cc:439] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.659677 7865 tablet_bootstrap.cc:439] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.659849 7865 ts_tablet_manager.cc:1059] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s I0720 00:44:32.660255 7871 tablet_bootstrap.cc:590] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.660558 7865 raft_consensus.cc:340] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.660778 7865 raft_consensus.cc:366] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.660827 7865 raft_consensus.cc:705] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.653724 7873 tablet_bootstrap.cc:439] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.661078 7873 ts_tablet_manager.cc:1059] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s I0720 00:44:32.661006 7865 consensus_queue.cc:229] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.661548 7865 ts_tablet_manager.cc:1087] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0720 00:44:32.661650 7873 raft_consensus.cc:340] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.661818 7865 tablet_bootstrap.cc:439] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.661890 7873 raft_consensus.cc:366] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.661988 7873 raft_consensus.cc:705] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.662966 7868 tablet_bootstrap.cc:439] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.663117 7865 tablet_bootstrap.cc:590] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.663166 7868 ts_tablet_manager.cc:1059] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s I0720 00:44:32.663868 7868 raft_consensus.cc:340] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.664162 7868 raft_consensus.cc:366] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.664222 7868 raft_consensus.cc:705] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.664466 7866 tablet_bootstrap.cc:439] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.664430 7868 consensus_queue.cc:229] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.664613 7866 ts_tablet_manager.cc:1059] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s I0720 00:44:32.665236 7868 ts_tablet_manager.cc:1087] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0720 00:44:32.665275 7866 raft_consensus.cc:340] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.665519 7868 tablet_bootstrap.cc:439] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda: Bootstrap starting. I0720 00:44:32.665511 7866 raft_consensus.cc:366] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.665572 7866 raft_consensus.cc:705] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.665796 7866 consensus_queue.cc:229] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.666477 7869 tablet_bootstrap.cc:439] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.666640 7869 ts_tablet_manager.cc:1059] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s I0720 00:44:32.667327 7869 raft_consensus.cc:340] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.667625 7869 raft_consensus.cc:366] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.667724 7869 raft_consensus.cc:705] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.667950 7869 consensus_queue.cc:229] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.668623 7869 ts_tablet_manager.cc:1087] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0720 00:44:32.654444 7872 raft_consensus.cc:340] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.669539 7872 raft_consensus.cc:366] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.669634 7872 raft_consensus.cc:705] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.669899 7872 consensus_queue.cc:229] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.672032 7872 ts_tablet_manager.cc:1087] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s I0720 00:44:32.672349 7872 tablet_bootstrap.cc:439] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.673797 7872 tablet_bootstrap.cc:590] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.674901 7870 tablet_bootstrap.cc:439] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.675045 7870 ts_tablet_manager.cc:1059] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s I0720 00:44:32.675591 7870 raft_consensus.cc:340] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.675859 7870 raft_consensus.cc:366] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.675959 7870 raft_consensus.cc:705] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.676298 7870 consensus_queue.cc:229] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.676487 7868 tablet_bootstrap.cc:590] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda: No blocks or log segments found. Creating new log. I0720 00:44:32.676786 7873 consensus_queue.cc:229] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.677685 7873 ts_tablet_manager.cc:1087] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s I0720 00:44:32.677852 7866 ts_tablet_manager.cc:1087] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s I0720 00:44:32.677963 7873 tablet_bootstrap.cc:439] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.677949 7867 tablet_bootstrap.cc:439] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.678221 7867 ts_tablet_manager.cc:1059] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.003s I0720 00:44:32.679077 7867 raft_consensus.cc:340] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.679270 7867 raft_consensus.cc:366] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.679642 7867 raft_consensus.cc:705] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.678408 7866 tablet_bootstrap.cc:439] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.680586 7867 consensus_queue.cc:229] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.681859 7866 tablet_bootstrap.cc:590] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.681861 7873 tablet_bootstrap.cc:590] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.682330 7674 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.682677 7674 raft_consensus.cc:2350] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.677042 7870 ts_tablet_manager.cc:1087] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0720 00:44:32.683341 7483 leader_election.cc:310] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab; no voters: I0720 00:44:32.683888 7883 raft_consensus.cc:2644] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.683966 7871 tablet_bootstrap.cc:439] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.683972 7883 raft_consensus.cc:472] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.684036 7883 raft_consensus.cc:2890] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.684103 7871 ts_tablet_manager.cc:1059] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.002s I0720 00:44:32.684751 7871 raft_consensus.cc:340] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.684963 7871 raft_consensus.cc:366] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.685003 7871 raft_consensus.cc:705] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.685189 7871 consensus_queue.cc:229] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.685406 7886 raft_consensus.cc:472] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.686578 7868 tablet_bootstrap.cc:439] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda: No bootstrap required, opened a new log I0720 00:44:32.686446 7883 raft_consensus.cc:494] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.685593 7886 raft_consensus.cc:494] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.687059 7883 leader_election.cc:296] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.686981 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.687544 7798 raft_consensus.cc:2350] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.687575 7674 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.687721 7674 raft_consensus.cc:2890] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.687957 7886 leader_election.cc:296] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.688122 7868 ts_tablet_manager.cc:1059] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.005s I0720 00:44:32.688565 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.688724 7798 raft_consensus.cc:2890] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.690960 7866 tablet_bootstrap.cc:439] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.691141 7866 ts_tablet_manager.cc:1059] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s I0720 00:44:32.691349 7798 raft_consensus.cc:2350] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.691560 7868 raft_consensus.cc:340] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.691781 7868 raft_consensus.cc:366] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.691766 7481 leader_election.cc:310] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.691871 7868 raft_consensus.cc:705] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Initialized, Role: FOLLOWER I0720 00:44:32.691879 7866 raft_consensus.cc:340] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.692121 7868 consensus_queue.cc:229] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.692116 7883 raft_consensus.cc:2644] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.693169 7883 raft_consensus.cc:667] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Becoming Leader. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Running, Role: LEADER I0720 00:44:32.693589 7883 consensus_queue.cc:206] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.694265 7868 ts_tablet_manager.cc:1087] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s I0720 00:44:32.698545 7865 tablet_bootstrap.cc:439] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.698711 7865 ts_tablet_manager.cc:1059] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.001s I0720 00:44:32.700031 7865 raft_consensus.cc:340] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.700269 7865 raft_consensus.cc:366] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.700358 7865 raft_consensus.cc:705] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.699898 7424 catalog_manager.cc:4067] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f reported cstate change: term changed from 0 to 1, leader changed from to 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129). New cstate: current_term: 1 leader_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:32.700583 7865 consensus_queue.cc:229] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.701494 7865 ts_tablet_manager.cc:1087] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0720 00:44:32.701726 7865 tablet_bootstrap.cc:439] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Bootstrap starting. I0720 00:44:32.693040 7867 ts_tablet_manager.cc:1087] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s I0720 00:44:32.703253 7866 raft_consensus.cc:366] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.703346 7866 raft_consensus.cc:705] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.704074 7865 tablet_bootstrap.cc:590] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No blocks or log segments found. Creating new log. I0720 00:44:32.704516 7866 consensus_queue.cc:229] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.705127 7866 ts_tablet_manager.cc:1087] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s I0720 00:44:32.706363 7865 tablet_bootstrap.cc:439] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: No bootstrap required, opened a new log I0720 00:44:32.706506 7865 ts_tablet_manager.cc:1059] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I0720 00:44:32.707160 7865 raft_consensus.cc:340] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.707415 7865 raft_consensus.cc:366] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.707486 7865 raft_consensus.cc:705] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Initialized, Role: FOLLOWER I0720 00:44:32.707640 7871 ts_tablet_manager.cc:1087] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s I0720 00:44:32.707700 7865 consensus_queue.cc:229] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.707784 7887 raft_consensus.cc:472] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.707937 7887 raft_consensus.cc:494] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.709434 7887 leader_election.cc:296] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.710680 7674 raft_consensus.cc:2350] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.710803 7865 ts_tablet_manager.cc:1087] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s I0720 00:44:32.715474 7871 tablet_bootstrap.cc:439] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab: Bootstrap starting. I0720 00:44:32.718729 7872 tablet_bootstrap.cc:439] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.718876 7872 ts_tablet_manager.cc:1059] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.047s user 0.000s sys 0.003s I0720 00:44:32.719660 7872 raft_consensus.cc:340] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.719867 7872 raft_consensus.cc:366] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.719923 7872 raft_consensus.cc:705] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.720099 7872 consensus_queue.cc:229] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.720528 7872 ts_tablet_manager.cc:1087] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0720 00:44:32.720728 7871 tablet_bootstrap.cc:590] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab: No blocks or log segments found. Creating new log. I0720 00:44:32.722230 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4d36dab0fb81459dbc2df72f66852ea2" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.722358 7873 tablet_bootstrap.cc:439] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.722437 7798 raft_consensus.cc:2350] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 0. I0720 00:44:32.722471 7873 ts_tablet_manager.cc:1059] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s I0720 00:44:32.723525 7606 leader_election.cc:310] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.724090 7886 raft_consensus.cc:2644] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.724380 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a487d770322e4226b00934877a3bff5d" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:32.724619 7886 raft_consensus.cc:472] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.724649 7549 raft_consensus.cc:2350] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 0. I0720 00:44:32.724716 7886 raft_consensus.cc:2890] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.725263 7732 leader_election.cc:310] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.726114 7887 raft_consensus.cc:2644] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.726189 7887 raft_consensus.cc:472] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.726238 7887 raft_consensus.cc:2890] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.727392 7873 raft_consensus.cc:340] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.727618 7873 raft_consensus.cc:366] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.728334 7887 raft_consensus.cc:494] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.728471 7873 raft_consensus.cc:705] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.728476 7886 raft_consensus.cc:494] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.728711 7873 consensus_queue.cc:229] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.728885 7887 leader_election.cc:296] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.729147 7886 leader_election.cc:296] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.729630 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a487d770322e4226b00934877a3bff5d" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:32.729794 7549 raft_consensus.cc:2890] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.731287 7873 ts_tablet_manager.cc:1087] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s I0720 00:44:32.731892 7549 raft_consensus.cc:2350] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. I0720 00:44:32.732553 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4d36dab0fb81459dbc2df72f66852ea2" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.732728 7798 raft_consensus.cc:2890] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.733850 7871 tablet_bootstrap.cc:439] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab: No bootstrap required, opened a new log I0720 00:44:32.734011 7871 ts_tablet_manager.cc:1059] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.001s I0720 00:44:32.734794 7798 raft_consensus.cc:2350] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:32.734719 7871 raft_consensus.cc:340] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.734922 7871 raft_consensus.cc:366] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0720 00:44:32.734975 7883 raft_consensus.cc:472] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.734979 7871 raft_consensus.cc:705] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Initialized, Role: FOLLOWER I0720 00:44:32.735170 7732 leader_election.cc:310] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.735178 7871 consensus_queue.cc:229] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.735182 7883 raft_consensus.cc:494] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.735466 7887 raft_consensus.cc:2644] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.735743 7883 leader_election.cc:296] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.736196 7606 leader_election.cc:310] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.736268 7674 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a487d770322e4226b00934877a3bff5d" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.736435 7886 raft_consensus.cc:2644] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.736778 7674 raft_consensus.cc:2350] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 0. I0720 00:44:32.737010 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "a487d770322e4226b00934877a3bff5d" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.737228 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.737263 7672 raft_consensus.cc:2890] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.737386 7798 raft_consensus.cc:2350] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.736588 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.737125 7886 raft_consensus.cc:667] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Becoming Leader. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Running, Role: LEADER I0720 00:44:32.737746 7481 leader_election.cc:310] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.738150 7883 raft_consensus.cc:2644] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.738201 7883 raft_consensus.cc:472] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.738108 7673 raft_consensus.cc:2350] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.738265 7883 raft_consensus.cc:2890] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.740049 7887 raft_consensus.cc:667] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 1 LEADER]: Becoming Leader. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Running, Role: LEADER I0720 00:44:32.740058 7883 raft_consensus.cc:494] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.740383 7887 consensus_queue.cc:206] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.740634 7883 leader_election.cc:296] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.740722 7886 consensus_queue.cc:206] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.740968 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.741152 7673 raft_consensus.cc:2890] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.741891 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.742195 7672 raft_consensus.cc:2350] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. I0720 00:44:32.742054 7798 raft_consensus.cc:2890] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.742612 7548 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4d36dab0fb81459dbc2df72f66852ea2" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:32.742794 7548 raft_consensus.cc:2890] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.743232 7673 raft_consensus.cc:2350] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.743577 7483 leader_election.cc:310] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab; no voters: I0720 00:44:32.743846 7883 raft_consensus.cc:2644] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.743966 7883 raft_consensus.cc:667] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Becoming Leader. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Running, Role: LEADER I0720 00:44:32.744156 7883 consensus_queue.cc:206] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.744621 7798 raft_consensus.cc:2350] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.744817 7548 raft_consensus.cc:2350] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:32.745489 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "4d36dab0fb81459dbc2df72f66852ea2" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:32.745242 7424 catalog_manager.cc:4067] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda reported cstate change: term changed from 0 to 1, leader changed from to c3780073b3594c88b678ec4da2bd8cda (127.0.58.131). New cstate: current_term: 1 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:32.745718 7549 raft_consensus.cc:2263] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. Re-sending same reply. I0720 00:44:32.746882 7871 ts_tablet_manager.cc:1087] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s I0720 00:44:32.747066 7426 catalog_manager.cc:4067] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f reported cstate change: term changed from 0 to 1, leader changed from to 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129). New cstate: current_term: 1 leader_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:32.749827 7424 catalog_manager.cc:4067] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab reported cstate change: term changed from 0 to 1, leader changed from to 7a005754141d43ac91f33e8797a489ab (127.0.58.130). New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:32.805809 7886 raft_consensus.cc:472] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.805984 7886 raft_consensus.cc:494] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.806623 7886 leader_election.cc:296] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.807252 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd37441ffc54486b90ba5230a81da533" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:32.807514 7549 raft_consensus.cc:2350] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 0. I0720 00:44:32.807528 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd37441ffc54486b90ba5230a81da533" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.807727 7798 raft_consensus.cc:2350] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 0. I0720 00:44:32.807881 7608 leader_election.cc:310] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab; no voters: I0720 00:44:32.808202 7886 raft_consensus.cc:2644] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.808262 7886 raft_consensus.cc:472] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.808326 7886 raft_consensus.cc:2890] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.809763 7883 raft_consensus.cc:472] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.809882 7883 raft_consensus.cc:494] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.810124 7886 raft_consensus.cc:494] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.810575 7883 leader_election.cc:296] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.810865 7886 leader_election.cc:296] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.810955 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.811014 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd37441ffc54486b90ba5230a81da533" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:32.811051 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.811172 7549 raft_consensus.cc:2890] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.811159 7798 raft_consensus.cc:2350] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.811213 7673 raft_consensus.cc:2350] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.811492 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "bd37441ffc54486b90ba5230a81da533" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.811552 7481 leader_election.cc:310] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.811666 7797 raft_consensus.cc:2890] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.811887 7883 raft_consensus.cc:2644] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.811969 7883 raft_consensus.cc:472] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.812041 7883 raft_consensus.cc:2890] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.813483 7549 raft_consensus.cc:2350] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:32.813702 7797 raft_consensus.cc:2350] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:32.813835 7608 leader_election.cc:310] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab; no voters: I0720 00:44:32.813771 7883 raft_consensus.cc:494] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.814123 7886 raft_consensus.cc:2644] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.814278 7886 raft_consensus.cc:667] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Becoming Leader. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Running, Role: LEADER I0720 00:44:32.814497 7886 consensus_queue.cc:206] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.814617 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.814754 7797 raft_consensus.cc:2890] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.814729 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.814905 7673 raft_consensus.cc:2890] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.816195 7797 raft_consensus.cc:2350] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.816542 7481 leader_election.cc:310] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.816814 7884 raft_consensus.cc:2644] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.816920 7884 raft_consensus.cc:667] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Becoming Leader. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Running, Role: LEADER I0720 00:44:32.817157 7673 raft_consensus.cc:2350] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.817343 7883 leader_election.cc:296] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Requested vote from peers c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.817175 7884 consensus_queue.cc:206] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:32.817991 7424 catalog_manager.cc:4067] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab reported cstate change: term changed from 0 to 1, leader changed from to 7a005754141d43ac91f33e8797a489ab (127.0.58.130). New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:32.820289 7426 catalog_manager.cc:4067] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f reported cstate change: term changed from 0 to 1, leader changed from to 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129). New cstate: current_term: 1 leader_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: HEALTHY } } } I0720 00:44:32.858543 7887 raft_consensus.cc:472] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.858716 7887 raft_consensus.cc:494] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.859278 7887 leader_election.cc:296] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.859966 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "eb86205d3d8548f4b0daaa602f89c3a9" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.859966 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "eb86205d3d8548f4b0daaa602f89c3a9" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:32.860172 7673 raft_consensus.cc:2350] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 0. I0720 00:44:32.860177 7549 raft_consensus.cc:2350] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 0. I0720 00:44:32.860566 7732 leader_election.cc:310] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.860810 7887 raft_consensus.cc:2644] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.860868 7887 raft_consensus.cc:472] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.860932 7887 raft_consensus.cc:2890] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.862498 7887 raft_consensus.cc:494] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.862916 7887 leader_election.cc:296] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:32.863417 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "eb86205d3d8548f4b0daaa602f89c3a9" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:32.863432 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "eb86205d3d8548f4b0daaa602f89c3a9" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.863595 7549 raft_consensus.cc:2890] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.863596 7673 raft_consensus.cc:2890] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.865691 7549 raft_consensus.cc:2350] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. I0720 00:44:32.865691 7673 raft_consensus.cc:2350] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. I0720 00:44:32.866053 7732 leader_election.cc:310] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.866552 7887 raft_consensus.cc:2644] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.866688 7887 raft_consensus.cc:667] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 1 LEADER]: Becoming Leader. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Running, Role: LEADER I0720 00:44:32.866894 7887 consensus_queue.cc:206] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } W0720 00:44:32.868422 7593 tablet.cc:1799] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: Can't schedule compaction. Clean time has not been advanced past its initial value. I0720 00:44:32.869931 7426 catalog_manager.cc:4067] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: term changed from 0 to 1, leader changed from to c3780073b3594c88b678ec4da2bd8cda (127.0.58.131). New cstate: current_term: 1 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:32.871050 7884 raft_consensus.cc:472] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.871179 7884 raft_consensus.cc:494] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.871760 7884 leader_election.cc:296] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.872205 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.872221 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.872395 7797 raft_consensus.cc:2350] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.872411 7673 raft_consensus.cc:2350] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.872763 7481 leader_election.cc:310] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.873020 7884 raft_consensus.cc:2644] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.873075 7884 raft_consensus.cc:472] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.873128 7884 raft_consensus.cc:2890] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.874656 7884 raft_consensus.cc:494] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.875120 7884 leader_election.cc:296] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.875540 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.875537 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.875694 7797 raft_consensus.cc:2890] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.875708 7673 raft_consensus.cc:2890] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.877751 7673 raft_consensus.cc:2350] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.877760 7797 raft_consensus.cc:2350] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.878136 7483 leader_election.cc:310] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab; no voters: I0720 00:44:32.878247 7884 raft_consensus.cc:472] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.878422 7885 raft_consensus.cc:2644] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.878352 7884 raft_consensus.cc:494] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.878588 7885 raft_consensus.cc:667] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Becoming Leader. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Running, Role: LEADER I0720 00:44:32.878892 7885 consensus_queue.cc:206] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.879211 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:32.879375 7673 raft_consensus.cc:2350] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.879382 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.879561 7797 raft_consensus.cc:2350] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 0. I0720 00:44:32.879878 7481 leader_election.cc:310] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.880110 7884 leader_election.cc:296] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.880431 7883 raft_consensus.cc:2644] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.880542 7883 raft_consensus.cc:472] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.880627 7883 raft_consensus.cc:2890] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.881402 7426 catalog_manager.cc:4067] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f reported cstate change: term changed from 0 to 1, leader changed from to 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129). New cstate: current_term: 1 leader_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:32.882266 7883 raft_consensus.cc:494] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.882748 7883 leader_election.cc:296] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.883131 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.883146 7673 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:32.883292 7797 raft_consensus.cc:2890] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.883363 7673 raft_consensus.cc:2890] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.885303 7797 raft_consensus.cc:2350] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.885316 7673 raft_consensus.cc:2350] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 110cfca4c9df4d82bf4a2e9abadfcf5f in term 1. I0720 00:44:32.885619 7481 leader_election.cc:310] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.885836 7883 raft_consensus.cc:2644] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.885941 7883 raft_consensus.cc:667] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Becoming Leader. State: Replica: 110cfca4c9df4d82bf4a2e9abadfcf5f, State: Running, Role: LEADER I0720 00:44:32.886139 7883 consensus_queue.cc:206] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.888172 7426 catalog_manager.cc:4067] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f reported cstate change: term changed from 0 to 1, leader changed from to 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129). New cstate: current_term: 1 leader_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:32.933938 7886 raft_consensus.cc:472] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0720 00:44:32.934137 7886 raft_consensus.cc:494] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.934763 7886 leader_election.cc:296] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.935302 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9a5c9555b3fa4c92940250278686a179" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:32.935322 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9a5c9555b3fa4c92940250278686a179" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:32.935511 7549 raft_consensus.cc:2350] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 0. I0720 00:44:32.935513 7797 raft_consensus.cc:2350] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 0. I0720 00:44:32.935890 7608 leader_election.cc:310] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab; no voters: I0720 00:44:32.936148 7886 raft_consensus.cc:2644] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Leader pre-election won for term 1 I0720 00:44:32.936221 7886 raft_consensus.cc:472] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0720 00:44:32.936281 7886 raft_consensus.cc:2890] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.937919 7886 raft_consensus.cc:494] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.938485 7886 leader_election.cc:296] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:32.938926 7549 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9a5c9555b3fa4c92940250278686a179" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:32.938957 7797 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "9a5c9555b3fa4c92940250278686a179" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:32.939105 7797 raft_consensus.cc:2890] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.939141 7549 raft_consensus.cc:2890] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 0 FOLLOWER]: Advancing to term 1 I0720 00:44:32.940621 7797 raft_consensus.cc:2350] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:32.940739 7549 raft_consensus.cc:2350] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:32.941027 7606 leader_election.cc:310] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: I0720 00:44:32.941282 7886 raft_consensus.cc:2644] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader election won for term 1 I0720 00:44:32.941385 7886 raft_consensus.cc:667] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Becoming Leader. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Running, Role: LEADER I0720 00:44:32.941565 7886 consensus_queue.cc:206] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:32.944169 7426 catalog_manager.cc:4067] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab reported cstate change: term changed from 0 to 1, leader changed from to 7a005754141d43ac91f33e8797a489ab (127.0.58.130). New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:33.095557 7797 raft_consensus.cc:1184] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.096695 7886 consensus_queue.cc:984] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.099495 7798 raft_consensus.cc:1184] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.099541 7549 raft_consensus.cc:1184] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.100201 7886 consensus_queue.cc:984] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.100549 7907 consensus_queue.cc:984] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.101397 7549 raft_consensus.cc:1184] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.101845 7907 consensus_queue.cc:984] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.116926 7796 raft_consensus.cc:1184] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.117811 7547 raft_consensus.cc:1184] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.123663 7886 consensus_queue.cc:984] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s I0720 00:44:33.124856 7886 consensus_queue.cc:984] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.140774 7673 raft_consensus.cc:1184] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.141294 7674 raft_consensus.cc:1184] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.145947 7672 raft_consensus.cc:1184] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.147014 7674 raft_consensus.cc:1184] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.143157 7883 consensus_queue.cc:984] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s I0720 00:44:33.147408 7885 consensus_queue.cc:984] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s I0720 00:44:33.151017 7884 consensus_queue.cc:984] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s I0720 00:44:33.153676 7887 consensus_queue.cc:984] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s I0720 00:44:33.155673 7795 raft_consensus.cc:1184] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.156415 7794 raft_consensus.cc:1184] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.156637 7797 raft_consensus.cc:1184] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.162729 7673 raft_consensus.cc:1184] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.164736 7673 raft_consensus.cc:1184] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.166793 7671 raft_consensus.cc:1184] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.171202 7917 consensus_queue.cc:984] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.172884 7545 raft_consensus.cc:1184] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I0720 00:44:33.173369 7885 consensus_queue.cc:984] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s I0720 00:44:33.173763 7883 consensus_queue.cc:984] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s I0720 00:44:33.174178 7546 raft_consensus.cc:1184] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.174679 7885 consensus_queue.cc:984] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.177407 7917 consensus_queue.cc:984] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.178602 7917 consensus_queue.cc:984] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.199174 7883 consensus_queue.cc:984] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0720 00:44:33.199755 7883 consensus_queue.cc:984] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.200860 7796 raft_consensus.cc:1184] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.207327 7883 consensus_queue.cc:984] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.212330 7949 mvcc.cc:203] Tried to move safe_time back from 6404437905989894144 to 6404437904864944128. Current Snapshot: MvccSnapshot[committed={T|T < 6404437905911754752}] I0720 00:44:33.231057 7798 raft_consensus.cc:1184] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 110cfca4c9df4d82bf4a2e9abadfcf5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0720 00:44:33.233780 7944 mvcc.cc:203] Tried to move safe_time back from 6404437905989894144 to 6404437904864944128. Current Snapshot: MvccSnapshot[committed={T|T < 6404437905989894144}] I0720 00:44:33.234108 7885 consensus_queue.cc:984] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0720 00:44:33.257382 7936 mvcc.cc:203] Tried to move safe_time back from 6404437905781403648 to 6404437905170419712. Current Snapshot: MvccSnapshot[committed={T|T < 6404437905781403648}] W0720 00:44:33.448570 7481 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.131:37319, user_credentials={real_user=slave}} blocked reactor thread for 75652.1us I0720 00:44:33.630446 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000279 I0720 00:44:33.638619 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000282 I0720 00:44:33.640511 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000282 I0720 00:44:33.749996 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.118s user 0.016s sys 0.010s Metrics: {"bytes_written":6048,"cfile_init":1,"compiler_manager_pool.queue_time_us":1190,"data dirs.queue_time_us":392,"data dirs.run_cpu_time_us":165,"data dirs.run_wall_time_us":557,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1651,"lbm_writes_lt_1ms":23,"rows_written":54,"thread_start_us":2430,"threads_started":2} I0720 00:44:33.751816 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 636 bytes on disk I0720 00:44:33.752660 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:33.753533 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000312 I0720 00:44:33.758754 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.120s user 0.039s sys 0.022s Metrics: {"bytes_written":5651,"cfile_init":1,"compiler_manager_pool.queue_time_us":5356,"data dirs.queue_time_us":35179,"data dirs.run_cpu_time_us":204,"data dirs.run_wall_time_us":12261,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"rows_written":39,"thread_start_us":44050,"threads_started":2} I0720 00:44:33.762498 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 486 bytes on disk I0720 00:44:33.763325 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4} I0720 00:44:33.775629 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000319 I0720 00:44:33.811760 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.036s user 0.015s sys 0.000s Metrics: {"bytes_written":6806,"cfile_init":1,"data dirs.queue_time_us":885,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":715,"lbm_writes_lt_1ms":23,"rows_written":81,"thread_start_us":3056,"threads_started":1} I0720 00:44:33.812881 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 884 bytes on disk I0720 00:44:33.813503 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:33.814708 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000329 I0720 00:44:33.826900 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.050s user 0.012s sys 0.001s Metrics: {"bytes_written":6243,"cfile_init":1,"data dirs.queue_time_us":422,"data dirs.run_cpu_time_us":146,"data dirs.run_wall_time_us":7114,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":740,"lbm_writes_lt_1ms":23,"rows_written":61,"thread_start_us":3893,"threads_started":1} I0720 00:44:33.828469 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 684 bytes on disk I0720 00:44:33.828507 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.014s user 0.010s sys 0.001s Metrics: {"bytes_written":6197,"cfile_init":1,"data dirs.queue_time_us":6595,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":616,"lbm_writes_lt_1ms":23,"rows_written":58} I0720 00:44:33.829566 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 653 bytes on disk I0720 00:44:33.830544 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:33.834213 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000327 I0720 00:44:33.850122 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.016s user 0.007s sys 0.005s Metrics: {"bytes_written":6886,"cfile_init":1,"data dirs.queue_time_us":2781,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"rows_written":85} I0720 00:44:33.851310 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 927 bytes on disk I0720 00:44:33.851877 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:33.852676 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000328 I0720 00:44:33.869891 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.015s user 0.013s sys 0.001s Metrics: {"bytes_written":6812,"cfile_init":1,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"rows_written":81} I0720 00:44:33.871070 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 894 bytes on disk I0720 00:44:33.874608 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4} I0720 00:44:33.875458 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000333 I0720 00:44:33.907049 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4} I0720 00:44:33.909381 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000355 I0720 00:44:33.953143 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.077s user 0.018s sys 0.009s Metrics: {"bytes_written":7098,"cfile_init":1,"data dirs.queue_time_us":35021,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":660,"lbm_writes_lt_1ms":23,"rows_written":95,"thread_start_us":36355,"threads_started":1} I0720 00:44:33.953393 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.312s user 0.016s sys 0.009s Metrics: {"bytes_written":6041,"cfile_init":1,"compiler_manager_pool.queue_time_us":1900,"data dirs.queue_time_us":1177,"data dirs.run_cpu_time_us":177,"data dirs.run_wall_time_us":24351,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":22796,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":54,"thread_start_us":86343,"threads_started":2} I0720 00:44:33.955262 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 636 bytes on disk I0720 00:44:33.956851 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:33.966596 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000372 I0720 00:44:33.989774 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 1010 bytes on disk I0720 00:44:33.999325 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.009s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:34.011718 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.102s user 0.014s sys 0.002s Metrics: {"bytes_written":7087,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":121,"data dirs.run_wall_time_us":470,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"rows_written":94} I0720 00:44:34.012831 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 1001 bytes on disk I0720 00:44:34.013353 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:34.014133 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000377 I0720 00:44:34.023237 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000366 I0720 00:44:34.053822 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.039s user 0.011s sys 0.003s Metrics: {"bytes_written":7662,"cfile_init":1,"data dirs.queue_time_us":1656,"data dirs.run_cpu_time_us":138,"data dirs.run_wall_time_us":1002,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2948,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":115,"thread_start_us":3557,"threads_started":1} I0720 00:44:34.055073 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 1211 bytes on disk I0720 00:44:34.055675 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:34.056443 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000387 I0720 00:44:34.073349 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":7428,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":126,"data dirs.run_wall_time_us":590,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":673,"lbm_writes_lt_1ms":23,"rows_written":105} I0720 00:44:34.074645 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 1121 bytes on disk I0720 00:44:34.075281 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4} I0720 00:44:34.076108 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000391 I0720 00:44:34.153558 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.077s user 0.014s sys 0.001s Metrics: {"bytes_written":7716,"cfile_init":1,"data dirs.queue_time_us":83,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":527,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":661,"lbm_writes_lt_1ms":23,"rows_written":118} I0720 00:44:34.179358 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000413 W0720 00:44:34.187708 7732 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.129:42263, user_credentials={real_user=slave}} blocked reactor thread for 49203.9us I0720 00:44:34.199375 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.020s user 0.012s sys 0.003s Metrics: {"bytes_written":8292,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":129,"data dirs.run_wall_time_us":596,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":743,"lbm_writes_lt_1ms":23,"rows_written":139} I0720 00:44:34.200600 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000416 I0720 00:44:34.216485 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":8274,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":134,"data dirs.run_wall_time_us":632,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"rows_written":138} I0720 00:44:34.217794 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000420 I0720 00:44:34.219918 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.252s user 0.015s sys 0.002s Metrics: {"bytes_written":7423,"cfile_init":1,"data dirs.queue_time_us":28152,"data dirs.run_cpu_time_us":201,"data dirs.run_wall_time_us":1989,"drs_written":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"rows_written":105,"thread_start_us":35120,"threads_started":1} I0720 00:44:34.274878 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000457 I0720 00:44:34.289475 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.266s user 0.014s sys 0.000s Metrics: {"bytes_written":7527,"cfile_init":1,"data dirs.queue_time_us":56,"data dirs.run_cpu_time_us":145,"data dirs.run_wall_time_us":627,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"rows_written":111} I0720 00:44:34.340265 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000450 I0720 00:44:34.359782 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.142s user 0.009s sys 0.007s Metrics: {"bytes_written":7520,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":203,"data dirs.run_wall_time_us":948,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"rows_written":112} I0720 00:44:34.361057 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000458 I0720 00:44:34.409411 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.069s user 0.012s sys 0.003s Metrics: {"bytes_written":8538,"cfile_init":1,"data dirs.queue_time_us":5708,"data dirs.run_cpu_time_us":155,"data dirs.run_wall_time_us":699,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4536,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":149} I0720 00:44:34.415072 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 1543 bytes on disk I0720 00:44:34.415766 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4} I0720 00:44:34.416612 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 1184 bytes on disk I0720 00:44:34.417160 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:34.417877 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000469 I0720 00:44:34.435499 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.074s user 0.010s sys 0.003s Metrics: {"bytes_written":8760,"cfile_init":1,"data dirs.queue_time_us":68,"data dirs.run_cpu_time_us":158,"data dirs.run_wall_time_us":570,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463,"lbm_writes_lt_1ms":23,"rows_written":160,"spinlock_wait_cycles":1920} I0720 00:44:34.436455 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 1630 bytes on disk I0720 00:44:34.436975 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:34.437615 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 1447 bytes on disk I0720 00:44:34.438159 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4} I0720 00:44:34.438769 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 1441 bytes on disk I0720 00:44:34.439227 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4} I0720 00:44:34.439772 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 1256 bytes on disk I0720 00:44:34.440246 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4} I0720 00:44:34.440827 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 1186 bytes on disk I0720 00:44:34.441234 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4} I0720 00:44:34.460877 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.043s user 0.015s sys 0.000s Metrics: {"bytes_written":8565,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":145,"data dirs.run_wall_time_us":892,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":752,"lbm_writes_lt_1ms":23,"rows_written":151} I0720 00:44:34.467039 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 1557 bytes on disk I0720 00:44:34.467679 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:34.468513 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000481 I0720 00:44:34.506517 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.038s user 0.013s sys 0.005s Metrics: {"bytes_written":8760,"cfile_init":1,"data dirs.queue_time_us":62,"data dirs.run_cpu_time_us":136,"data dirs.run_wall_time_us":759,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"rows_written":160} I0720 00:44:34.507833 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 1630 bytes on disk I0720 00:44:34.508533 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4} I0720 00:44:34.689569 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.414s user 0.015s sys 0.002s Metrics: {"bytes_written":7648,"cfile_init":1,"data dirs.queue_time_us":62,"data dirs.run_cpu_time_us":156,"data dirs.run_wall_time_us":574,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"rows_written":114} I0720 00:44:34.709373 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000574 I0720 00:44:34.742372 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.032s user 0.015s sys 0.007s Metrics: {"bytes_written":9721,"cfile_init":1,"data dirs.queue_time_us":5313,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"rows_written":194,"thread_start_us":5484,"threads_started":1} I0720 00:44:34.743546 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000582 I0720 00:44:34.786449 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000278 I0720 00:44:34.853582 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.110s user 0.013s sys 0.004s Metrics: {"bytes_written":9902,"cfile_init":1,"data dirs.queue_time_us":1709,"data dirs.run_cpu_time_us":156,"data dirs.run_wall_time_us":8210,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"rows_written":201,"thread_start_us":8260,"threads_started":1} I0720 00:44:34.873188 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000616 W0720 00:44:34.882987 7606 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.131:37319, user_credentials={real_user=slave}} blocked reactor thread for 103742us I0720 00:44:34.948927 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.141s user 0.021s sys 0.000s Metrics: {"bytes_written":9297,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":125,"data dirs.run_wall_time_us":655,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":889,"lbm_writes_lt_1ms":23,"rows_written":177} I0720 00:44:34.957940 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.949639 I0720 00:44:34.960628 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000336 I0720 00:44:34.991961 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.031s user 0.018s sys 0.001s Metrics: {"bytes_written":9530,"cfile_init":1,"data dirs.queue_time_us":683,"data dirs.run_cpu_time_us":126,"data dirs.run_wall_time_us":6748,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"rows_written":188,"thread_start_us":2649,"threads_started":1} I0720 00:44:34.993739 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.981143 I0720 00:44:35.004987 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.131s user 0.014s sys 0.003s Metrics: {"bytes_written":10118,"cfile_init":1,"data dirs.queue_time_us":12217,"data dirs.run_cpu_time_us":137,"data dirs.run_wall_time_us":713,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"rows_written":209} I0720 00:44:35.012647 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000652 I0720 00:44:35.047001 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.053s user 0.024s sys 0.001s Metrics: {"bytes_written":10959,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4629,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13583,"cfile_init":6,"data dirs.queue_time_us":5018,"data dirs.run_cpu_time_us":501,"data dirs.run_wall_time_us":4218,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":926,"lbm_reads_lt_1ms":50,"lbm_write_time_us":767,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":242,"thread_start_us":10825,"threads_started":2} I0720 00:44:35.048524 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000346 I0720 00:44:35.058770 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.093s user 0.029s sys 0.004s Metrics: {"bytes_written":10232,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4157,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12293,"cfile_init":6,"data dirs.queue_time_us":16772,"data dirs.run_cpu_time_us":609,"data dirs.run_wall_time_us":3724,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1127,"lbm_reads_lt_1ms":50,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":216,"spinlock_wait_cycles":768,"thread_start_us":13954,"threads_started":2} I0720 00:44:35.060164 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000347 I0720 00:44:35.093173 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.044s user 0.015s sys 0.000s Metrics: {"bytes_written":8762,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":125,"data dirs.run_wall_time_us":496,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":777,"lbm_writes_lt_1ms":23,"rows_written":159} I0720 00:44:35.102551 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 2415 bytes on disk I0720 00:44:35.107664 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4} I0720 00:44:35.108639 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.986399 I0720 00:44:35.129818 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.021s user 0.016s sys 0.004s Metrics: {"bytes_written":10372,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4230,"cfile_cache_miss":26,"cfile_cache_miss_bytes":12422,"cfile_init":6,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":1417,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":860,"lbm_reads_lt_1ms":50,"lbm_write_time_us":755,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":220} I0720 00:44:35.131150 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 2214 bytes on disk I0720 00:44:35.150604 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:35.154922 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000317 I0720 00:44:35.175693 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.163s user 0.017s sys 0.007s Metrics: {"bytes_written":10959,"cfile_init":1,"data dirs.queue_time_us":3892,"data dirs.run_cpu_time_us":144,"data dirs.run_wall_time_us":721,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":721,"lbm_writes_lt_1ms":23,"rows_written":242} I0720 00:44:35.189045 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000700 I0720 00:44:35.204546 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.144s user 0.011s sys 0.005s Metrics: {"bytes_written":8711,"cfile_init":1,"data dirs.queue_time_us":48,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":505,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2325,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":156} I0720 00:44:35.241850 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.052s user 0.011s sys 0.011s Metrics: {"bytes_written":11597,"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":144,"data dirs.run_wall_time_us":557,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":772,"lbm_writes_lt_1ms":23,"rows_written":267} I0720 00:44:35.243142 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000713 I0720 00:44:35.265362 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533): perf score=1.003533 I0720 00:44:35.266404 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.023s user 0.021s sys 0.002s Metrics: {"bytes_written":11218,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":135,"data dirs.run_wall_time_us":649,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":741,"lbm_writes_lt_1ms":23,"rows_written":255} I0720 00:44:35.267536 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000715 I0720 00:44:35.314043 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.046s user 0.022s sys 0.000s Metrics: {"bytes_written":11474,"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":127,"data dirs.run_wall_time_us":565,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"rows_written":264} I0720 00:44:35.379668 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000396 I0720 00:44:35.411315 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.146s user 0.022s sys 0.000s Metrics: {"bytes_written":10788,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4539,"cfile_cache_miss":26,"cfile_cache_miss_bytes":13491,"cfile_init":6,"data dirs.queue_time_us":8733,"data dirs.run_cpu_time_us":614,"data dirs.run_wall_time_us":1081,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":69313,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":722,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":237} I0720 00:44:35.415374 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000441 W0720 00:44:35.423647 7732 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:40487, user_credentials={real_user=slave}} blocked reactor thread for 47951.7us I0720 00:44:35.519616 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.073s user 0.018s sys 0.000s Metrics: {"bytes_written":9968,"cfile_init":1,"data dirs.queue_time_us":60,"data dirs.run_cpu_time_us":145,"data dirs.run_wall_time_us":499,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"rows_written":205} I0720 00:44:35.529286 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.984241 I0720 00:44:35.447086 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.292s user 0.019s sys 0.000s Metrics: {"bytes_written":8745,"cfile_init":1,"data dirs.queue_time_us":68,"data dirs.run_cpu_time_us":160,"data dirs.run_wall_time_us":955,"drs_written":1,"lbm_read_time_us":21880,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":26771,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":158} I0720 00:44:35.538105 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 1651 bytes on disk I0720 00:44:35.551450 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4} I0720 00:44:35.560150 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533): perf score=1.003934 I0720 00:44:35.598786 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.219s user 0.021s sys 0.001s Metrics: {"bytes_written":10131,"cfile_init":1,"data dirs.queue_time_us":539,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"rows_written":210,"thread_start_us":4295,"threads_started":1} I0720 00:44:35.601938 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 2680 bytes on disk I0720 00:44:35.602589 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:35.603425 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.998738 I0720 00:44:35.600483 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.071s user 0.023s sys 0.000s Metrics: {"bytes_written":11450,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5016,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14780,"cfile_init":6,"data dirs.queue_time_us":5658,"data dirs.run_cpu_time_us":556,"data dirs.run_wall_time_us":983,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1017,"lbm_reads_lt_1ms":50,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":263} I0720 00:44:35.604564 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000487 I0720 00:44:35.777158 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.174s user 0.022s sys 0.005s Metrics: {"bytes_written":11451,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5033,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14831,"cfile_init":6,"data dirs.queue_time_us":45711,"data dirs.run_cpu_time_us":660,"data dirs.run_wall_time_us":1085,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":5307,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":709,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":264} I0720 00:44:35.795059 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000437 I0720 00:44:35.806094 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.197s user 0.018s sys 0.000s Metrics: {"bytes_written":10364,"cfile_init":1,"data dirs.queue_time_us":54928,"data dirs.run_cpu_time_us":165,"data dirs.run_wall_time_us":1121,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1901,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":219} I0720 00:44:35.807498 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=1.002964 I0720 00:44:35.818619 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.258s user 0.018s sys 0.004s Metrics: {"bytes_written":11186,"cfile_cache_hit":10,"cfile_cache_hit_bytes":4814,"cfile_cache_miss":26,"cfile_cache_miss_bytes":14124,"cfile_init":5,"data dirs.queue_time_us":11141,"data dirs.run_cpu_time_us":536,"data dirs.run_wall_time_us":1414,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":688,"lbm_reads_lt_1ms":46,"lbm_write_time_us":769,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":252,"thread_start_us":16691,"threads_started":2} I0720 00:44:35.820211 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 2544 bytes on disk I0720 00:44:35.820816 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:35.821596 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000491 I0720 00:44:35.850383 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.043s user 0.023s sys 0.001s Metrics: {"bytes_written":12491,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5763,"cfile_cache_miss":26,"cfile_cache_miss_bytes":17063,"cfile_init":6,"data dirs.queue_time_us":334,"data dirs.run_cpu_time_us":579,"data dirs.run_wall_time_us":1038,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":879,"lbm_reads_lt_1ms":50,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":304} I0720 00:44:35.851768 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000550 I0720 00:44:35.875538 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.023s user 0.022s sys 0.000s Metrics: {"bytes_written":11003,"cfile_init":1,"data dirs.queue_time_us":50,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":542,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":766,"lbm_writes_lt_1ms":23,"rows_written":245} I0720 00:44:35.876940 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 3034 bytes on disk I0720 00:44:35.877549 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:35.878540 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 2623 bytes on disk I0720 00:44:35.879071 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:35.879086 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.054s user 0.011s sys 0.008s Metrics: {"bytes_written":10143,"cfile_init":1,"data dirs.queue_time_us":1267,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":531,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4775,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":211} I0720 00:44:35.879810 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 2459 bytes on disk I0720 00:44:35.880399 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 2132 bytes on disk I0720 00:44:35.880391 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:35.881125 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 2397 bytes on disk I0720 00:44:35.881723 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:35.882531 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 2167 bytes on disk I0720 00:44:35.883093 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:35.883489 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:35.883848 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.977269 I0720 00:44:35.884327 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.987756 I0720 00:44:35.917045 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.033s user 0.018s sys 0.008s Metrics: {"bytes_written":13069,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6167,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18137,"cfile_init":5,"data dirs.queue_time_us":312,"data dirs.run_cpu_time_us":548,"data dirs.run_wall_time_us":1041,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":971,"lbm_reads_lt_1ms":46,"lbm_write_time_us":733,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":326} I0720 00:44:35.918414 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000546 I0720 00:44:35.942735 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.024s user 0.018s sys 0.005s Metrics: {"bytes_written":11573,"cfile_init":1,"data dirs.queue_time_us":63,"data dirs.run_cpu_time_us":112,"data dirs.run_wall_time_us":527,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":696,"lbm_writes_lt_1ms":23,"rows_written":266} I0720 00:44:35.944057 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=1.012523 I0720 00:44:35.950531 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.066s user 0.020s sys 0.005s Metrics: {"bytes_written":13069,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6162,"cfile_cache_miss":26,"cfile_cache_miss_bytes":18102,"cfile_init":5,"data dirs.queue_time_us":341,"data dirs.run_cpu_time_us":545,"data dirs.run_wall_time_us":1035,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":912,"lbm_reads_lt_1ms":46,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":326} I0720 00:44:35.951812 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 3216 bytes on disk I0720 00:44:35.952342 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:35.953121 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000522 I0720 00:44:35.975370 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.022s user 0.012s sys 0.008s Metrics: {"bytes_written":11133,"cfile_init":1,"data dirs.queue_time_us":42,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":527,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":758,"lbm_writes_lt_1ms":23,"rows_written":249} I0720 00:44:35.976806 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 2538 bytes on disk I0720 00:44:35.977423 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:35.978214 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179): perf score=1.008703 I0720 00:44:36.005218 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.027s user 0.025s sys 0.000s Metrics: {"bytes_written":13840,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6678,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19730,"cfile_init":5,"data dirs.queue_time_us":294,"data dirs.run_cpu_time_us":509,"data dirs.run_wall_time_us":929,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":964,"lbm_reads_lt_1ms":46,"lbm_write_time_us":1103,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":354} I0720 00:44:36.006588 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000515 I0720 00:44:36.023658 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.079s user 0.024s sys 0.000s Metrics: {"bytes_written":14021,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6803,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19881,"cfile_init":6,"data dirs.queue_time_us":25704,"data dirs.run_cpu_time_us":585,"data dirs.run_wall_time_us":1037,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":935,"lbm_reads_lt_1ms":50,"lbm_write_time_us":680,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":361} I0720 00:44:36.025068 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 3552 bytes on disk I0720 00:44:36.025676 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:36.026490 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000482 I0720 00:44:36.029896 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.023s user 0.013s sys 0.004s Metrics: {"bytes_written":10756,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":531,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"rows_written":236} I0720 00:44:36.031143 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 3486 bytes on disk I0720 00:44:36.031733 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I0720 00:44:36.032500 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 2371 bytes on disk I0720 00:44:36.033071 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:36.033795 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2): perf score=1.009528 I0720 00:44:36.053742 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.027s user 0.015s sys 0.004s Metrics: {"bytes_written":10966,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":614,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":773,"lbm_writes_lt_1ms":23,"rows_written":243} I0720 00:44:36.055120 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 3216 bytes on disk I0720 00:44:36.055634 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4} I0720 00:44:36.056397 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 2438 bytes on disk I0720 00:44:36.056942 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:36.057670 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2): perf score=1.009499 I0720 00:44:36.080713 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.023s user 0.018s sys 0.004s Metrics: {"bytes_written":13833,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6673,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19525,"cfile_init":5,"data dirs.queue_time_us":314,"data dirs.run_cpu_time_us":532,"data dirs.run_wall_time_us":883,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":883,"lbm_reads_lt_1ms":46,"lbm_write_time_us":668,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":354} I0720 00:44:36.081475 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.047s user 0.017s sys 0.007s Metrics: {"bytes_written":13833,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6673,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19493,"cfile_init":5,"data dirs.queue_time_us":335,"data dirs.run_cpu_time_us":566,"data dirs.run_wall_time_us":933,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":968,"lbm_reads_lt_1ms":46,"lbm_write_time_us":840,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":354} I0720 00:44:36.082211 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000465 I0720 00:44:36.082898 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 3478 bytes on disk I0720 00:44:36.083427 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:36.084123 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000523 I0720 00:44:36.096777 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.014s user 0.011s sys 0.000s Metrics: {"bytes_written":9800,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":122,"data dirs.run_wall_time_us":592,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"rows_written":199} I0720 00:44:36.098167 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 3478 bytes on disk I0720 00:44:36.098773 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:36.099570 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(a487d770322e4226b00934877a3bff5d): perf score=1.009732 I0720 00:44:36.119374 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.035s user 0.018s sys 0.000s Metrics: {"bytes_written":10125,"cfile_init":1,"data dirs.queue_time_us":56,"data dirs.run_cpu_time_us":104,"data dirs.run_wall_time_us":506,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":585,"lbm_writes_lt_1ms":23,"rows_written":209} I0720 00:44:36.120746 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 2138 bytes on disk I0720 00:44:36.123123 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4} I0720 00:44:36.124006 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(a487d770322e4226b00934877a3bff5d): perf score=1.009515 I0720 00:44:36.150450 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.026s user 0.017s sys 0.008s Metrics: {"bytes_written":13656,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6566,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19396,"cfile_init":5,"data dirs.queue_time_us":303,"data dirs.run_cpu_time_us":526,"data dirs.run_wall_time_us":951,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":907,"lbm_reads_lt_1ms":46,"lbm_write_time_us":824,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":348} I0720 00:44:36.151829 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000537 I0720 00:44:36.172304 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.020s user 0.012s sys 0.005s Metrics: {"bytes_written":10209,"cfile_init":1,"data dirs.queue_time_us":45,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":523,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"rows_written":215} I0720 00:44:36.173652 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 3444 bytes on disk I0720 00:44:36.174681 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4} I0720 00:44:36.175523 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=1.009902 I0720 00:44:36.224910 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.049s user 0.009s sys 0.017s Metrics: {"bytes_written":13826,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6656,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19568,"cfile_init":6,"data dirs.queue_time_us":399,"data dirs.run_cpu_time_us":557,"data dirs.run_wall_time_us":997,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":921,"lbm_reads_lt_1ms":50,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":353,"spinlock_wait_cycles":2304} I0720 00:44:36.226450 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 3481 bytes on disk I0720 00:44:36.227059 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:36.227941 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000519 I0720 00:44:36.255394 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.027s user 0.021s sys 0.000s Metrics: {"bytes_written":11619,"cfile_init":1,"data dirs.queue_time_us":61,"data dirs.run_cpu_time_us":123,"data dirs.run_wall_time_us":560,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"rows_written":270} I0720 00:44:36.257020 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=1.012597 I0720 00:44:36.289294 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.031s user 0.028s sys 0.000s Metrics: {"bytes_written":14503,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7185,"cfile_cache_miss":26,"cfile_cache_miss_bytes":20857,"cfile_init":6,"data dirs.queue_time_us":592,"data dirs.run_cpu_time_us":686,"data dirs.run_wall_time_us":1130,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1986,"lbm_reads_lt_1ms":50,"lbm_write_time_us":645,"lbm_writes_lt_1ms":23,"mutex_wait_us":20,"num_input_rowsets":2,"rows_written":382,"spinlock_wait_cycles":26368} I0720 00:44:36.290899 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000515 I0720 00:44:36.322520 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.031s user 0.018s sys 0.002s Metrics: {"bytes_written":10612,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":509,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"rows_written":231} I0720 00:44:36.323964 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=1.011754 W0720 00:44:36.382536 7606 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.131:37319, user_credentials={real_user=slave}} blocked reactor thread for 99199.6us I0720 00:44:36.405673 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.610s user 0.014s sys 0.004s Metrics: {"bytes_written":9569,"cfile_init":1,"data dirs.queue_time_us":58341,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":630,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"rows_written":190} I0720 00:44:36.421133 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 3048 bytes on disk I0720 00:44:36.422367 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":179,"lbm_reads_lt_1ms":8} I0720 00:44:36.423246 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 2663 bytes on disk I0720 00:44:36.423897 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:36.424821 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.990730 I0720 00:44:36.538982 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.114s user 0.024s sys 0.015s Metrics: {"bytes_written":12273,"cfile_cache_hit":10,"cfile_cache_hit_bytes":5600,"cfile_cache_miss":26,"cfile_cache_miss_bytes":16450,"cfile_init":5,"data dirs.queue_time_us":44523,"data dirs.run_cpu_time_us":524,"data dirs.run_wall_time_us":7006,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":895,"lbm_reads_lt_1ms":46,"lbm_write_time_us":7655,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"rows_written":295,"thread_start_us":52643,"threads_started":2} I0720 00:44:36.540715 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 2921 bytes on disk I0720 00:44:36.541298 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:36.542279 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 2647 bytes on disk I0720 00:44:36.542785 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:36.543656 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000515 I0720 00:44:36.561650 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.235s user 0.019s sys 0.011s Metrics: {"bytes_written":14722,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7347,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21285,"cfile_init":6,"data dirs.queue_time_us":4644,"data dirs.run_cpu_time_us":646,"data dirs.run_wall_time_us":1076,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1219,"lbm_reads_lt_1ms":50,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":391} I0720 00:44:36.563087 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000436 I0720 00:44:36.609846 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.066s user 0.019s sys 0.005s Metrics: {"bytes_written":12046,"cfile_init":1,"data dirs.queue_time_us":2237,"data dirs.run_cpu_time_us":120,"data dirs.run_wall_time_us":602,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3043,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"rows_written":285} I0720 00:44:36.613034 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 4066 bytes on disk I0720 00:44:36.613862 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":177,"lbm_reads_lt_1ms":8} I0720 00:44:36.649952 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179): perf score=1.008588 I0720 00:44:36.747293 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.097s user 0.022s sys 0.005s Metrics: {"bytes_written":14928,"cfile_cache_hit":10,"cfile_cache_hit_bytes":7499,"cfile_cache_miss":26,"cfile_cache_miss_bytes":21989,"cfile_init":5,"data dirs.queue_time_us":318,"data dirs.run_cpu_time_us":533,"data dirs.run_wall_time_us":1099,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":20658,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":44,"lbm_write_time_us":776,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":399} I0720 00:44:36.748605 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000557 I0720 00:44:36.762823 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.663s user 0.025s sys 0.001s Metrics: {"bytes_written":13656,"cfile_cache_hit":10,"cfile_cache_hit_bytes":6566,"cfile_cache_miss":26,"cfile_cache_miss_bytes":19240,"cfile_init":6,"data dirs.queue_time_us":1431,"data dirs.run_cpu_time_us":478,"data dirs.run_wall_time_us":1881,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1022,"lbm_reads_lt_1ms":50,"lbm_write_time_us":12229,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":21,"num_input_rowsets":2,"rows_written":348,"thread_start_us":1494,"threads_started":2} I0720 00:44:36.764858 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 3444 bytes on disk I0720 00:44:36.766790 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4} I0720 00:44:36.768664 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000641 I0720 00:44:36.771136 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.195s user 0.006s sys 0.013s Metrics: {"bytes_written":9806,"cfile_init":1,"data dirs.queue_time_us":2182,"data dirs.run_cpu_time_us":119,"data dirs.run_wall_time_us":584,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":34396,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"rows_written":200} I0720 00:44:36.772404 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec): perf score=1.012688 I0720 00:44:36.831549 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.059s user 0.024s sys 0.004s Metrics: {"bytes_written":16088,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8279,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24099,"cfile_init":6,"data dirs.queue_time_us":3952,"data dirs.run_cpu_time_us":637,"data dirs.run_wall_time_us":1038,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1052,"lbm_reads_lt_1ms":50,"lbm_write_time_us":779,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":442} I0720 00:44:36.832988 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000483 I0720 00:44:36.859191 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.024s user 0.014s sys 0.005s Metrics: {"bytes_written":10358,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":102,"data dirs.run_wall_time_us":685,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1059,"lbm_writes_lt_1ms":23,"rows_written":220} I0720 00:44:36.860498 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=1.009120 I0720 00:44:36.894294 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.031s user 0.019s sys 0.009s Metrics: {"bytes_written":15958,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8240,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24030,"cfile_init":6,"data dirs.queue_time_us":313,"data dirs.run_cpu_time_us":543,"data dirs.run_wall_time_us":1016,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1162,"lbm_reads_lt_1ms":50,"lbm_write_time_us":906,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":440} I0720 00:44:36.895627 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000402 I0720 00:44:36.909268 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.160s user 0.015s sys 0.006s Metrics: {"bytes_written":10993,"cfile_init":1,"data dirs.queue_time_us":4182,"data dirs.run_cpu_time_us":107,"data dirs.run_wall_time_us":611,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"rows_written":244} I0720 00:44:36.914443 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 4427 bytes on disk I0720 00:44:36.915380 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":169,"lbm_reads_lt_1ms":8} I0720 00:44:36.916268 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 3886 bytes on disk I0720 00:44:36.917286 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:36.918817 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533): perf score=1.006657 I0720 00:44:36.921044 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.023s user 0.017s sys 0.000s Metrics: {"bytes_written":9577,"cfile_init":1,"data dirs.queue_time_us":53,"data dirs.run_cpu_time_us":111,"data dirs.run_wall_time_us":513,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":705,"lbm_writes_lt_1ms":23,"rows_written":192} I0720 00:44:36.922567 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 4286 bytes on disk I0720 00:44:36.923214 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:36.924154 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 4272 bytes on disk I0720 00:44:36.924674 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:36.925770 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533): perf score=1.006296 I0720 00:44:36.968508 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.041s user 0.027s sys 0.000s Metrics: {"bytes_written":16113,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8313,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24179,"cfile_init":6,"data dirs.queue_time_us":350,"data dirs.run_cpu_time_us":551,"data dirs.run_wall_time_us":918,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":919,"lbm_reads_lt_1ms":50,"lbm_write_time_us":680,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":444} I0720 00:44:36.969889 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 4322 bytes on disk I0720 00:44:36.970801 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:36.971592 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 3810 bytes on disk I0720 00:44:36.972152 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4} I0720 00:44:36.972968 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000304 I0720 00:44:37.025115 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.106s user 0.025s sys 0.002s Metrics: {"bytes_written":15927,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8206,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24032,"cfile_init":5,"data dirs.queue_time_us":4857,"data dirs.run_cpu_time_us":631,"data dirs.run_wall_time_us":1108,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":838,"lbm_reads_lt_1ms":46,"lbm_write_time_us":750,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":438} I0720 00:44:37.029625 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 4259 bytes on disk I0720 00:44:37.032402 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4} I0720 00:44:37.053660 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.283s user 0.019s sys 0.003s Metrics: {"bytes_written":12272,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":128,"data dirs.run_wall_time_us":949,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":785,"lbm_writes_lt_1ms":23,"rows_written":294} I0720 00:44:37.057435 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179): perf score=1.009007 I0720 00:44:37.067505 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 2534 bytes on disk I0720 00:44:37.068176 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4} I0720 00:44:37.075430 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 2415 bytes on disk I0720 00:44:37.076035 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:37.084666 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 2106 bytes on disk I0720 00:44:37.085108 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4} I0720 00:44:37.110414 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 2025 bytes on disk I0720 00:44:37.120308 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.008s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4} I0720 00:44:37.121243 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000630 I0720 00:44:37.159835 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.102s user 0.023s sys 0.005s Metrics: {"bytes_written":16137,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8336,"cfile_cache_miss":26,"cfile_cache_miss_bytes":24282,"cfile_init":6,"data dirs.queue_time_us":2621,"data dirs.run_cpu_time_us":566,"data dirs.run_wall_time_us":1055,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":3393,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":852,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":445} I0720 00:44:37.161188 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 4297 bytes on disk I0720 00:44:37.161877 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4} I0720 00:44:37.162760 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000738 I0720 00:44:37.174254 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.201s user 0.014s sys 0.003s Metrics: {"bytes_written":8276,"cfile_init":1,"data dirs.queue_time_us":41,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":454,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":749,"lbm_writes_lt_1ms":23,"rows_written":139} I0720 00:44:37.175562 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 3758 bytes on disk I0720 00:44:37.176256 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:37.177093 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=1.001396 I0720 00:44:37.209317 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.032s user 0.028s sys 0.000s Metrics: {"bytes_written":16660,"cfile_cache_hit":10,"cfile_cache_hit_bytes":8701,"cfile_cache_miss":26,"cfile_cache_miss_bytes":25275,"cfile_init":6,"data dirs.queue_time_us":346,"data dirs.run_cpu_time_us":571,"data dirs.run_wall_time_us":954,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1049,"lbm_reads_lt_1ms":50,"lbm_write_time_us":855,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":465} I0720 00:44:37.210669 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000343 I0720 00:44:37.233050 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.022s user 0.010s sys 0.004s Metrics: {"bytes_written":8767,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":114,"data dirs.run_wall_time_us":509,"drs_written":1,"lbm_read_time_us":171,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"rows_written":159} I0720 00:44:37.234375 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179): perf score=1.013438 I0720 00:44:37.303984 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.141s user 0.019s sys 0.006s Metrics: {"bytes_written":13275,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":123,"data dirs.run_wall_time_us":656,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9740,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":21,"rows_written":336} I0720 00:44:37.305449 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=1.012514 I0720 00:44:37.313019 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.192s user 0.021s sys 0.003s Metrics: {"bytes_written":12382,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":113,"data dirs.run_wall_time_us":509,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"rows_written":298} W0720 00:44:37.315685 7483 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.58.130:40487, user_credentials={real_user=slave}} blocked reactor thread for 56623us I0720 00:44:37.316794 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 2965 bytes on disk I0720 00:44:37.317292 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4} I0720 00:44:37.318737 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(a487d770322e4226b00934877a3bff5d): perf score=1.014325 I0720 00:44:37.345532 234 external_mini_cluster.cc:938] Running /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-2 --block_manager=log --rpc_bind_addresses=127.0.58.132:0 --local_ip_for_outbound_sockets=127.0.58.132 --webserver_interface=127.0.58.132 --webserver_port=0 --tserver_master_addrs=127.0.58.190:34075 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --flush_threshold_secs=1 --flush_threshold_mb=1 with env {} I0720 00:44:37.349525 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.031s user 0.025s sys 0.004s Metrics: {"bytes_written":17511,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9324,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27180,"cfile_init":5,"data dirs.queue_time_us":804,"data dirs.run_cpu_time_us":551,"data dirs.run_wall_time_us":2210,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1047,"lbm_reads_lt_1ms":46,"lbm_write_time_us":898,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":499} I0720 00:44:37.350867 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000652 I0720 00:44:37.392938 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.087s user 0.029s sys 0.000s Metrics: {"bytes_written":17368,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9262,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26632,"cfile_init":6,"data dirs.queue_time_us":350,"data dirs.run_cpu_time_us":526,"data dirs.run_wall_time_us":984,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1810,"lbm_reads_lt_1ms":50,"lbm_write_time_us":771,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":496} I0720 00:44:37.393066 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.158s user 0.030s sys 0.001s Metrics: {"bytes_written":17869,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9577,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27855,"cfile_init":6,"data dirs.queue_time_us":2497,"data dirs.run_cpu_time_us":474,"data dirs.run_wall_time_us":874,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":17422,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":49,"lbm_write_time_us":798,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":513} I0720 00:44:37.394313 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000680 I0720 00:44:37.394449 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000379 I0720 00:44:37.405166 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.054s user 0.021s sys 0.000s Metrics: {"bytes_written":11850,"cfile_init":1,"data dirs.queue_time_us":1111,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":531,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":692,"lbm_writes_lt_1ms":23,"rows_written":280} I0720 00:44:37.407306 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 4823 bytes on disk I0720 00:44:37.410408 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.003s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":2517,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0720 00:44:37.411300 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=1.013334 I0720 00:44:37.413866 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.019s user 0.015s sys 0.000s Metrics: {"bytes_written":8765,"cfile_init":1,"data dirs.queue_time_us":229,"data dirs.run_cpu_time_us":127,"data dirs.run_wall_time_us":539,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":583,"lbm_writes_lt_1ms":23,"rows_written":159} I0720 00:44:37.415182 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2): perf score=1.013235 I0720 00:44:37.418464 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.024s user 0.023s sys 0.000s Metrics: {"bytes_written":13043,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":509,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":764,"lbm_writes_lt_1ms":23,"rows_written":324} I0720 00:44:37.419606 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 4776 bytes on disk I0720 00:44:37.420131 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:37.420862 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 3199 bytes on disk I0720 00:44:37.421341 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4} I0720 00:44:37.422009 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec): perf score=1.011637 I0720 00:44:37.445466 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.030s user 0.028s sys 0.000s Metrics: {"bytes_written":17904,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9577,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27823,"cfile_init":6,"data dirs.queue_time_us":349,"data dirs.run_cpu_time_us":487,"data dirs.run_wall_time_us":842,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":822,"lbm_reads_lt_1ms":50,"lbm_write_time_us":738,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":513} I0720 00:44:37.446772 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 4980 bytes on disk I0720 00:44:37.447351 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I0720 00:44:37.448225 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 4953 bytes on disk I0720 00:44:37.448933 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:37.449627 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 4494 bytes on disk I0720 00:44:37.450186 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:37.450942 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000370 I0720 00:44:37.452894 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.031s user 0.025s sys 0.004s Metrics: {"bytes_written":18547,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10066,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29174,"cfile_init":5,"data dirs.queue_time_us":315,"data dirs.run_cpu_time_us":435,"data dirs.run_wall_time_us":796,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":803,"lbm_reads_lt_1ms":46,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":540} I0720 00:44:37.454097 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000625 I0720 00:44:37.469135 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.018s user 0.014s sys 0.003s Metrics: {"bytes_written":9397,"cfile_init":1,"data dirs.queue_time_us":57,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":495,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":562,"lbm_writes_lt_1ms":23,"rows_written":184} I0720 00:44:37.470530 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 1873 bytes on disk I0720 00:44:37.471161 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4} I0720 00:44:37.471976 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(a487d770322e4226b00934877a3bff5d): perf score=1.014165 I0720 00:44:37.477239 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.023s user 0.019s sys 0.000s Metrics: {"bytes_written":11633,"cfile_init":1,"data dirs.queue_time_us":58,"data dirs.run_cpu_time_us":162,"data dirs.run_wall_time_us":585,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"rows_written":269} I0720 00:44:37.478521 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533): perf score=1.008957 I0720 00:44:37.502359 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.030s user 0.024s sys 0.004s Metrics: {"bytes_written":18359,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9920,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28778,"cfile_init":5,"data dirs.queue_time_us":353,"data dirs.run_cpu_time_us":500,"data dirs.run_wall_time_us":884,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":792,"lbm_reads_lt_1ms":46,"lbm_write_time_us":637,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":532} I0720 00:44:37.503739 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 5155 bytes on disk I0720 00:44:37.504494 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4} I0720 00:44:37.505282 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000370 I0720 00:44:37.505621 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.094s user 0.027s sys 0.001s Metrics: {"bytes_written":17309,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9138,"cfile_cache_miss":26,"cfile_cache_miss_bytes":26562,"cfile_init":6,"data dirs.queue_time_us":4894,"data dirs.run_cpu_time_us":472,"data dirs.run_wall_time_us":946,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":910,"lbm_reads_lt_1ms":50,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":489} I0720 00:44:37.506875 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000647 I0720 00:44:37.508834 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.030s user 0.023s sys 0.004s Metrics: {"bytes_written":17734,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9453,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27567,"cfile_init":6,"data dirs.queue_time_us":587,"data dirs.run_cpu_time_us":491,"data dirs.run_wall_time_us":900,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":955,"lbm_reads_lt_1ms":50,"lbm_write_time_us":607,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":506} I0720 00:44:37.509981 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000559 I0720 00:44:37.522428 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":8898,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":115,"data dirs.run_wall_time_us":519,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"rows_written":163} I0720 00:44:37.523666 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=1.011520 I0720 00:44:37.536733 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.027s user 0.012s sys 0.005s Metrics: {"bytes_written":10765,"cfile_init":1,"data dirs.queue_time_us":52,"data dirs.run_cpu_time_us":95,"data dirs.run_wall_time_us":553,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":633,"lbm_writes_lt_1ms":23,"rows_written":236} I0720 00:44:37.538134 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=1.013435 I0720 00:44:37.553843 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.030s user 0.025s sys 0.004s Metrics: {"bytes_written":17927,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9633,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28139,"cfile_init":6,"data dirs.queue_time_us":394,"data dirs.run_cpu_time_us":536,"data dirs.run_wall_time_us":941,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":924,"lbm_reads_lt_1ms":50,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":516} I0720 00:44:37.555133 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000361 I0720 00:44:37.559725 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.053s user 0.023s sys 0.001s Metrics: {"bytes_written":12407,"cfile_init":1,"data dirs.queue_time_us":2230,"data dirs.run_cpu_time_us":116,"data dirs.run_wall_time_us":545,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":700,"lbm_writes_lt_1ms":23,"rows_written":298} WARNING: Logging before InitGoogleLogging() is written to STDERR W0720 00:44:37.562531 8059 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0720 00:44:37.562922 8059 flags.cc:404] Enabled unsafe flag: --never_fsync=true I0720 00:44:37.563117 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 4756 bytes on disk I0720 00:44:37.563762 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4} I0720 00:44:37.564527 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 2968 bytes on disk I0720 00:44:37.565059 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4} I0720 00:44:37.565789 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec): perf score=1.011914 W0720 00:44:37.566212 8059 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0720 00:44:37.566370 8059 flags.cc:404] Enabled experimental flag: --flush_threshold_mb=1 W0720 00:44:37.566448 8059 flags.cc:404] Enabled experimental flag: --flush_threshold_secs=1 W0720 00:44:37.566558 8059 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.58.132 I0720 00:44:37.569723 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.031s user 0.028s sys 0.000s Metrics: {"bytes_written":17493,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9319,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27061,"cfile_init":6,"data dirs.queue_time_us":371,"data dirs.run_cpu_time_us":461,"data dirs.run_wall_time_us":841,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":970,"lbm_reads_lt_1ms":50,"lbm_write_time_us":747,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":499} I0720 00:44:37.570964 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 5252 bytes on disk I0720 00:44:37.571511 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:37.571599 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.016s user 0.012s sys 0.001s Metrics: {"bytes_written":8660,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":100,"data dirs.run_wall_time_us":467,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":584,"lbm_writes_lt_1ms":23,"rows_written":153} I0720 00:44:37.572280 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 4933 bytes on disk I0720 00:44:37.572926 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=1.013728 I0720 00:44:37.572888 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4} I0720 00:44:37.573545 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000492 I0720 00:44:37.581295 8059 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-2 --fs_wal_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.0.58.132:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb --webserver_interface=127.0.58.132 --webserver_port=0 --flush_threshold_mb=1 --flush_threshold_secs=1 --tserver_master_addrs=127.0.58.190:34075 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.0.58.132 --log_dir=/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.11.0-cdh6.x-SNAPSHOT revision aae905889e37d4b4704dffdd32555586ef5ab337 build type FASTDEBUG built by jenkins at 19 Jul 2019 16:58:05 PST on shared-ubuntu1404-ec2-m4-xlarge-ondemand-19ae.vpc.cloudera.com build id 551 ASAN enabled I0720 00:44:37.591866 8059 system_ntp.cc:193] NTP initialized. Skew: 500ppm Current error: 100016us I0720 00:44:37.592165 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.018s user 0.012s sys 0.003s Metrics: {"bytes_written":10161,"cfile_init":1,"data dirs.queue_time_us":46,"data dirs.run_cpu_time_us":83,"data dirs.run_wall_time_us":431,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"rows_written":212} I0720 00:44:37.592859 8059 fs_manager.cc:263] Metadata directory not provided I0720 00:44:37.593009 8059 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0720 00:44:37.593473 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=1.013290 I0720 00:44:37.593472 8059 server_base.cc:448] Could not load existing FS layout: Not found: could not find a healthy instance file I0720 00:44:37.593559 8059 server_base.cc:449] Attempting to create new FS layout instead I0720 00:44:37.596516 8059 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/instance: uuid: "f828308d73154e0ebda8ef74a1ce2e08" format_stamp: "Formatted at 2019-07-20 00:44:37 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:37.597328 8059 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-1/instance: uuid: "f828308d73154e0ebda8ef74a1ce2e08" format_stamp: "Formatted at 2019-07-20 00:44:37 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:37.597831 8059 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-2/instance: uuid: "f828308d73154e0ebda8ef74a1ce2e08" format_stamp: "Formatted at 2019-07-20 00:44:37 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:37.598333 8059 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/wal/instance: uuid: "f828308d73154e0ebda8ef74a1ce2e08" format_stamp: "Formatted at 2019-07-20 00:44:37 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:37.603849 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.031s user 0.025s sys 0.004s Metrics: {"bytes_written":18397,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9981,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29017,"cfile_init":6,"data dirs.queue_time_us":846,"data dirs.run_cpu_time_us":543,"data dirs.run_wall_time_us":909,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":925,"lbm_reads_lt_1ms":50,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"mutex_wait_us":25,"num_input_rowsets":2,"rows_written":535,"spinlock_wait_cycles":8320} I0720 00:44:37.605075 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000356 I0720 00:44:37.617483 8059 fs_manager.cc:503] Time spent creating directory manager: real 0.019s user 0.009s sys 0.007s I0720 00:44:37.619385 8059 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I0720 00:44:37.620096 8059 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I0720 00:44:37.620625 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":8077,"cfile_init":1,"data dirs.queue_time_us":51,"data dirs.run_cpu_time_us":125,"data dirs.run_wall_time_us":524,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"rows_written":132,"spinlock_wait_cycles":2944} I0720 00:44:37.621857 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=1.007391 I0720 00:44:37.624106 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.030s user 0.028s sys 0.000s Metrics: {"bytes_written":17927,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9627,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28027,"cfile_init":6,"data dirs.queue_time_us":322,"data dirs.run_cpu_time_us":395,"data dirs.run_wall_time_us":761,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":848,"lbm_reads_lt_1ms":50,"lbm_write_time_us":634,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":516} I0720 00:44:37.625293 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 5024 bytes on disk I0720 00:44:37.625949 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I0720 00:44:37.626683 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 4837 bytes on disk I0720 00:44:37.627182 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4} I0720 00:44:37.627836 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000487 I0720 00:44:37.627924 8059 fs_manager.cc:419] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s I0720 00:44:37.628032 8059 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-1,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-2,/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/wal uuid: "f828308d73154e0ebda8ef74a1ce2e08" format_stamp: "Formatted at 2019-07-20 00:44:37 on dist-test-slave-dist-test-slave-lkfw" I0720 00:44:37.628315 8059 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/wal metadata directory: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/wal 3 data directories: /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-1/data, /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-2/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0720 00:44:37.631093 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.065s user 0.028s sys 0.001s Metrics: {"bytes_written":18547,"cfile_cache_hit":10,"cfile_cache_hit_bytes":10071,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29383,"cfile_init":5,"data dirs.queue_time_us":6608,"data dirs.run_cpu_time_us":518,"data dirs.run_wall_time_us":1087,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":899,"lbm_reads_lt_1ms":46,"lbm_write_time_us":684,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":540} I0720 00:44:37.632400 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000664 I0720 00:44:37.645642 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":9518,"cfile_init":1,"data dirs.queue_time_us":44,"data dirs.run_cpu_time_us":85,"data dirs.run_wall_time_us":517,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":684,"lbm_writes_lt_1ms":23,"rows_written":186} I0720 00:44:37.647001 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=1.002495 I0720 00:44:37.654119 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.032s user 0.027s sys 0.004s Metrics: {"bytes_written":18105,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9757,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28193,"cfile_init":6,"data dirs.queue_time_us":565,"data dirs.run_cpu_time_us":557,"data dirs.run_wall_time_us":936,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":911,"lbm_reads_lt_1ms":50,"lbm_write_time_us":676,"lbm_writes_lt_1ms":23,"mutex_wait_us":31,"num_input_rowsets":2,"rows_written":523} I0720 00:44:37.655562 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 5178 bytes on disk I0720 00:44:37.657135 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":858,"lbm_reads_lt_1ms":4} I0720 00:44:37.657795 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 5042 bytes on disk I0720 00:44:37.658272 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4} I0720 00:44:37.658875 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 5024 bytes on disk I0720 00:44:37.659307 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4} I0720 00:44:37.672969 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.040s user 0.012s sys 0.007s Metrics: {"bytes_written":11003,"cfile_init":1,"data dirs.queue_time_us":43,"data dirs.run_cpu_time_us":99,"data dirs.run_wall_time_us":508,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"rows_written":246} I0720 00:44:37.677047 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.030s user 0.024s sys 0.003s Metrics: {"bytes_written":17768,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9560,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27812,"cfile_init":6,"data dirs.queue_time_us":290,"data dirs.run_cpu_time_us":408,"data dirs.run_wall_time_us":779,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":903,"lbm_reads_lt_1ms":50,"lbm_write_time_us":678,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":512} I0720 00:44:37.678164 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 4932 bytes on disk I0720 00:44:37.678697 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4} I0720 00:44:37.679111 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2): perf score=1.013642 I0720 00:44:37.679375 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000482 I0720 00:44:37.696624 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":9152,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":90,"data dirs.run_wall_time_us":438,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"rows_written":174} I0720 00:44:37.697780 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 1803 bytes on disk I0720 00:44:37.698374 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:37.699056 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=1.013748 I0720 00:44:37.716120 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.037s user 0.028s sys 0.001s Metrics: {"bytes_written":17904,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9577,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27937,"cfile_init":6,"data dirs.queue_time_us":334,"data dirs.run_cpu_time_us":471,"data dirs.run_wall_time_us":829,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":926,"lbm_reads_lt_1ms":50,"lbm_write_time_us":689,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":513} I0720 00:44:37.717340 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 5252 bytes on disk I0720 00:44:37.717941 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I0720 00:44:37.718732 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000681 I0720 00:44:37.722720 8059 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I0720 00:44:37.730196 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.031s user 0.028s sys 0.000s Metrics: {"bytes_written":18397,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9981,"cfile_cache_miss":26,"cfile_cache_miss_bytes":29033,"cfile_init":5,"data dirs.queue_time_us":310,"data dirs.run_cpu_time_us":450,"data dirs.run_wall_time_us":812,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":821,"lbm_reads_lt_1ms":46,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":535} I0720 00:44:37.731412 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 5178 bytes on disk I0720 00:44:37.732015 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:37.732739 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(0d5f343fde1d404598e962001ef17ae2): perf score=0.000466 I0720 00:44:37.735296 8059 ts_tablet_manager.cc:357] Loaded tablet metadata (0 total tablets, 0 live tablets) I0720 00:44:37.735404 8059 ts_tablet_manager.cc:377] Registered 0 tablets I0720 00:44:37.741017 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.022s user 0.003s sys 0.016s Metrics: {"bytes_written":11580,"cfile_init":1,"data dirs.queue_time_us":47,"data dirs.run_cpu_time_us":87,"data dirs.run_wall_time_us":621,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":667,"lbm_writes_lt_1ms":23,"rows_written":268} I0720 00:44:37.742346 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 4980 bytes on disk I0720 00:44:37.742936 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4} I0720 00:44:37.743702 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 2691 bytes on disk I0720 00:44:37.744246 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:37.744948 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=1.011700 I0720 00:44:37.748996 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.016s user 0.010s sys 0.003s Metrics: {"bytes_written":8765,"cfile_init":1,"data dirs.queue_time_us":60,"data dirs.run_cpu_time_us":92,"data dirs.run_wall_time_us":494,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"rows_written":159} I0720 00:44:37.750196 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 1645 bytes on disk I0720 00:44:37.750763 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:37.751416 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2): perf score=1.013235 I0720 00:44:37.775812 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.024s user 0.020s sys 0.004s Metrics: {"bytes_written":17904,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9577,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27823,"cfile_init":5,"data dirs.queue_time_us":71,"data dirs.run_cpu_time_us":103,"data dirs.run_wall_time_us":515,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":718,"lbm_reads_lt_1ms":46,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":513} I0720 00:44:37.777249 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000467 I0720 00:44:37.778517 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.033s user 0.026s sys 0.004s Metrics: {"bytes_written":18105,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9757,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28245,"cfile_init":5,"data dirs.queue_time_us":547,"data dirs.run_cpu_time_us":606,"data dirs.run_wall_time_us":972,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":889,"lbm_reads_lt_1ms":46,"lbm_write_time_us":884,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":523,"spinlock_wait_cycles":3328} I0720 00:44:37.779793 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 5042 bytes on disk I0720 00:44:37.780373 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4} I0720 00:44:37.781112 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=0.000685 I0720 00:44:37.791045 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"bytes_written":9397,"cfile_init":1,"data dirs.queue_time_us":248,"data dirs.run_cpu_time_us":120,"data dirs.run_wall_time_us":522,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"rows_written":184,"spinlock_wait_cycles":2944} I0720 00:44:37.792033 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(a487d770322e4226b00934877a3bff5d): perf score=1.014165 I0720 00:44:37.804010 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.023s user 0.012s sys 0.008s Metrics: {"bytes_written":11172,"cfile_init":1,"data dirs.queue_time_us":83,"data dirs.run_cpu_time_us":131,"data dirs.run_wall_time_us":557,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":683,"lbm_writes_lt_1ms":23,"rows_written":252} I0720 00:44:37.805307 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 2551 bytes on disk I0720 00:44:37.805941 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I0720 00:44:37.806679 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9): perf score=1.010651 I0720 00:44:37.814476 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.022s user 0.011s sys 0.010s Metrics: {"bytes_written":18359,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9920,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28778,"cfile_init":6,"data dirs.queue_time_us":999,"data dirs.run_cpu_time_us":549,"data dirs.run_wall_time_us":918,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":677,"lbm_reads_lt_1ms":50,"lbm_write_time_us":580,"lbm_writes_lt_1ms":23,"mutex_wait_us":407,"num_input_rowsets":2,"rows_written":532} I0720 00:44:37.815390 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 5155 bytes on disk I0720 00:44:37.815917 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:37.816447 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2): 4980 bytes on disk I0720 00:44:37.816946 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(0d5f343fde1d404598e962001ef17ae2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4} I0720 00:44:37.834465 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.028s user 0.023s sys 0.003s Metrics: {"bytes_written":17927,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9627,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28049,"cfile_init":5,"data dirs.queue_time_us":720,"data dirs.run_cpu_time_us":489,"data dirs.run_wall_time_us":888,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":772,"lbm_reads_lt_1ms":46,"lbm_write_time_us":590,"lbm_writes_lt_1ms":23,"mutex_wait_us":1,"num_input_rowsets":2,"rows_written":516,"spinlock_wait_cycles":10112} I0720 00:44:37.835494 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9): 5024 bytes on disk I0720 00:44:37.835966 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(eb86205d3d8548f4b0daaa602f89c3a9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4} I0720 00:44:37.836465 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=0.000622 I0720 00:44:37.853018 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":11639,"cfile_init":1,"data dirs.queue_time_us":72,"data dirs.run_cpu_time_us":115,"data dirs.run_wall_time_us":543,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"rows_written":271,"spinlock_wait_cycles":4352} I0720 00:44:37.854179 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 2717 bytes on disk I0720 00:44:37.854776 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4} I0720 00:44:37.855322 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2): perf score=1.013851 I0720 00:44:37.862351 8059 rpc_server.cc:205] RPC server started. Bound to: 127.0.58.132:36535 I0720 00:44:37.866370 8059 webserver.cc:374] Webserver started at http://127.0.58.132:46561/ using document root and password file I0720 00:44:37.867198 8059 server_base.cc:629] Dumped server information to /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb I0720 00:44:37.870173 234 external_mini_cluster.cc:1000] Started /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver as pid 8059 W0720 00:44:37.888319 7589 generic_service.cc:128] {username='slave'} at 127.0.0.1:51578 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:37.888432 7589 generic_service.cc:148] {username='slave'} at 127.0.0.1:51578 changed flags via RPC: enable_tablet_copy from 'true' to 'false' W0720 00:44:37.896623 7714 generic_service.cc:128] {username='slave'} at 127.0.0.1:48844 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:37.896750 7714 generic_service.cc:148] {username='slave'} at 127.0.0.1:48844 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:37.897012 8180 heartbeater.cc:324] Connected to a master server at 127.0.58.190:34075 I0720 00:44:37.897244 8180 heartbeater.cc:405] Registering TS with master... I0720 00:44:37.897817 8180 heartbeater.cc:454] Master 127.0.58.190:34075 requested a full tablet report, sending... I0720 00:44:37.899188 7423 ts_manager.cc:161] Registered new tserver with Master: f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535) I0720 00:44:37.899377 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.044s user 0.022s sys 0.004s Metrics: {"bytes_written":18397,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9976,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28888,"cfile_init":5,"data dirs.queue_time_us":670,"data dirs.run_cpu_time_us":547,"data dirs.run_wall_time_us":935,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":1814,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":45,"lbm_write_time_us":628,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":535,"spinlock_wait_cycles":1280} I0720 00:44:37.900151 7423 master_service.cc:277] Signed X509 certificate for tserver {username='slave'} at 127.0.58.132:39215 I0720 00:44:37.900522 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=0.000415 W0720 00:44:37.904150 7838 generic_service.cc:128] {username='slave'} at 127.0.0.1:41254 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:37.904273 7838 generic_service.cc:148] {username='slave'} at 127.0.0.1:41254 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:37.910713 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling FlushMRSOp(c8db08710314422cbd1b616d57ec4cec): perf score=0.000316 W0720 00:44:37.911901 8177 generic_service.cc:128] {username='slave'} at 127.0.0.1:52236 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:37.912050 8177 generic_service.cc:148] {username='slave'} at 127.0.0.1:52236 changed flags via RPC: enable_tablet_copy from 'true' to 'false' I0720 00:44:37.913471 234 test_util.cc:212] Using random seed: 633804656 I0720 00:44:37.920190 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.019s user 0.014s sys 0.000s Metrics: {"bytes_written":10327,"cfile_init":1,"data dirs.queue_time_us":64,"data dirs.run_cpu_time_us":116,"data dirs.run_wall_time_us":536,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":451,"lbm_writes_lt_1ms":23,"rows_written":217} I0720 00:44:37.921537 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806): perf score=1.010411 I0720 00:44:37.923890 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: FlushMRSOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.013s user 0.010s sys 0.000s Metrics: {"bytes_written":7249,"cfile_init":1,"data dirs.queue_time_us":65,"data dirs.run_cpu_time_us":105,"data dirs.run_wall_time_us":448,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":627,"lbm_writes_lt_1ms":23,"rows_written":98} I0720 00:44:37.925079 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec): 1068 bytes on disk I0720 00:44:37.925662 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: UndoDeltaBlockGCOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I0720 00:44:37.926436 7593 maintenance_manager.cc:327] P 110cfca4c9df4d82bf4a2e9abadfcf5f: Scheduling CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec): perf score=1.013537 I0720 00:44:37.928674 234 test_workload.cc:306] TestWorkload: Skipping table creation because table test-workload already exists W0720 00:44:37.929596 7589 generic_service.cc:128] {username='slave'} at 127.0.0.1:51578 forcing change of non-runtime-safe flag env_inject_eio_globs I0720 00:44:37.929720 7589 generic_service.cc:148] {username='slave'} at 127.0.0.1:51578 changed flags via RPC: env_inject_eio_globs from '*' to '/tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/**' W0720 00:44:37.931243 7589 generic_service.cc:128] {username='slave'} at 127.0.0.1:51578 forcing change of non-runtime-safe flag env_inject_eio I0720 00:44:37.931365 7589 generic_service.cc:148] {username='slave'} at 127.0.0.1:51578 changed flags via RPC: env_inject_eio from '0' to '1.0' I0720 00:44:37.937645 234 disk_failure-itest.cc:256] Currently has 0 failed tablets W0720 00:44:37.942905 7487 fault_injection.cc:44] FAULT INJECTION ENABLED! W0720 00:44:37.942955 7487 fault_injection.cc:45] THIS SERVER MAY CRASH! E0720 00:44:37.943079 7487 env_posix.cc:316] I/O error, context: INJECTED FAILURE E0720 00:44:37.943316 7487 data_dirs.cc:1145] Directory /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/data marked as failed I0720 00:44:37.943452 7487 ts_tablet_manager.cc:1468] Data dir 09c81302ef494594815f6b9bf5958d8f has 10 tablets I0720 00:44:37.943529 7487 ts_tablet_manager.cc:1475] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.943965 7487 ts_tablet_manager.cc:1475] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.944043 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:37.944200 8196 tablet_replica.cc:273] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.944494 8196 raft_consensus.cc:2147] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:37.944864 8196 raft_consensus.cc:2174] T 0380f4644f22404e9f2b80565c2d7a0b P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.945282 7487 ts_tablet_manager.cc:1475] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.945561 8197 tablet_replica.cc:273] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.945751 8197 raft_consensus.cc:2147] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:37.946616 8197 raft_consensus.cc:2174] T 0d5f343fde1d404598e962001ef17ae2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.947240 7487 ts_tablet_manager.cc:1475] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.947490 8198 tablet_replica.cc:273] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.947654 8198 raft_consensus.cc:2147] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:37.948315 7487 ts_tablet_manager.cc:1475] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.948958 7487 ts_tablet_manager.cc:1475] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.949189 7487 ts_tablet_manager.cc:1475] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.949707 7487 ts_tablet_manager.cc:1475] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.949856 7487 ts_tablet_manager.cc:1475] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.950026 7487 ts_tablet_manager.cc:1475] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: failing tablet I0720 00:44:37.950577 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.029s user 0.025s sys 0.003s Metrics: {"bytes_written":17768,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9560,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27828,"cfile_init":6,"data dirs.queue_time_us":1195,"data dirs.run_cpu_time_us":451,"data dirs.run_wall_time_us":930,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":802,"lbm_reads_lt_1ms":50,"lbm_write_time_us":702,"lbm_writes_lt_1ms":23,"mutex_wait_us":19,"num_input_rowsets":2,"rows_written":512} I0720 00:44:37.949062 8198 raft_consensus.cc:2174] T 4d36dab0fb81459dbc2df72f66852ea2 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.951692 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2): 5178 bytes on disk I0720 00:44:37.948982 8196 tablet_replica.cc:273] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.949582 8197 tablet_replica.cc:273] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.952365 8197 raft_consensus.cc:2147] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:37.952292 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(4d36dab0fb81459dbc2df72f66852ea2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4} I0720 00:44:37.952917 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806): 4932 bytes on disk W0720 00:44:37.952355 7487 cfile_writer.cc:508] Unable to append data of size 1837 at offset 16: IO error: INJECTED FAILURE: Input/output error (error 5) I0720 00:44:37.953337 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(5570d23334524c9a9e1bbbdf1c4c3806) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4} I0720 00:44:37.953810 8197 raft_consensus.cc:2174] T 7b95ae04955844828ac3ced5e85c9ed5 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.953936 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000373 W0720 00:44:37.954116 7487 cfile_writer.cc:430] Unable to append block to file: IO error: INJECTED FAILURE: Input/output error (error 5) I0720 00:44:37.954124 8196 raft_consensus.cc:2147] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Raft consensus shutting down. W0720 00:44:37.954201 7487 multi_column_writer.cc:125] Unable to Finish writer for column key INT32 NOT NULL: IO error: INJECTED FAILURE: Input/output error (error 5) I0720 00:44:37.954314 8196 raft_consensus.cc:2174] T 5570d23334524c9a9e1bbbdf1c4c3806 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! E0720 00:44:37.955452 7487 env_posix.cc:316] I/O error, context: INJECTED FAILURE W0720 00:44:37.955574 7487 ts_tablet_manager.cc:1462] Data directory is already marked failed. W0720 00:44:37.955641 7487 log_block_manager.cc:1308] Container /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/data/7d6433d3318e4651abb63f59088daab3 being marked read-only: IO error: unable to append block's metadata during close: Failed to append data to file: INJECTED FAILURE: Input/output error (error 5) W0720 00:44:37.955705 7487 log_block_manager.cc:1562] Failed to abort block 0000000000000308: IO error: unable to append block's metadata during close: Failed to append data to file: INJECTED FAILURE: Input/output error (error 5) E0720 00:44:37.956185 7487 env_posix.cc:316] I/O error, context: INJECTED FAILURE W0720 00:44:37.956267 7487 ts_tablet_manager.cc:1462] Data directory is already marked failed. I0720 00:44:37.953948 8198 tablet_replica.cc:273] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica W0720 00:44:37.962946 7487 log_block_manager.cc:1308] Container /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/data/0c750a4201584ce18bbfe39a30d5a909 being marked read-only: IO error: unable to append block's metadata during close: Failed to append data to file: INJECTED FAILURE: Input/output error (error 5) I0720 00:44:37.963100 8198 raft_consensus.cc:2147] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:37.963315 8198 raft_consensus.cc:2174] T 9a5c9555b3fa4c92940250278686a179 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.963385 8197 tablet_replica.cc:273] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.963325 8196 tablet_replica.cc:273] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica W0720 00:44:37.963572 7487 log_block_manager.cc:1562] Failed to abort block 0000000000000310: IO error: unable to append block's metadata during close: Failed to append data to file: INJECTED FAILURE: Input/output error (error 5) I0720 00:44:37.963639 8196 raft_consensus.cc:2147] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:37.963697 8197 raft_consensus.cc:2147] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:37.964154 8196 raft_consensus.cc:2174] T a487d770322e4226b00934877a3bff5d P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.964381 8197 raft_consensus.cc:2174] T bd37441ffc54486b90ba5230a81da533 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! W0720 00:44:37.965122 7487 tablet_mm_ops.cc:161] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: Compaction failed on c8db08710314422cbd1b616d57ec4cec: IO error: Failed to finish DRS writer: INJECTED FAILURE: Input/output error (error 5) I0720 00:44:37.966212 8198 tablet_replica.cc:273] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.966292 8197 tablet_replica.cc:273] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f: stopping tablet replica I0720 00:44:37.966661 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":7913,"cfile_init":1,"data dirs.queue_time_us":67,"data dirs.run_cpu_time_us":108,"data dirs.run_wall_time_us":506,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":438,"lbm_writes_lt_1ms":23,"rows_written":125} I0720 00:44:37.967012 8197 raft_consensus.cc:2147] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus shutting down. I0720 00:44:37.966369 7487 maintenance_manager.cc:532] P 110cfca4c9df4d82bf4a2e9abadfcf5f: CompactRowSetsOp(c8db08710314422cbd1b616d57ec4cec) complete. Timing: real 0.039s user 0.019s sys 0.004s Metrics: {"cfile_cache_miss":18,"cfile_cache_miss_bytes":19661,"cfile_init":4,"data dirs.queue_time_us":702,"data dirs.run_cpu_time_us":604,"data dirs.run_wall_time_us":2367,"delta_iterators_relevant":2,"lbm_read_time_us":560,"lbm_reads_lt_1ms":34,"lbm_write_time_us":231,"lbm_writes_lt_1ms":6,"mutex_wait_us":2096,"num_input_rowsets":2,"spinlock_wait_cycles":2816,"tablet-open.queue_time_us":66821,"tablet-open.run_cpu_time_us":11233,"tablet-open.run_wall_time_us":59133,"thread_start_us":2134,"threads_started":3} I0720 00:44:37.967142 8197 raft_consensus.cc:2174] T eb86205d3d8548f4b0daaa602f89c3a9 P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.967605 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 1326 bytes on disk I0720 00:44:37.967833 8198 raft_consensus.cc:2147] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 LEADER]: Raft consensus shutting down. I0720 00:44:37.967877 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:37.968089 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4} I0720 00:44:37.968343 8198 raft_consensus.cc:2174] T c8db08710314422cbd1b616d57ec4cec P 110cfca4c9df4d82bf4a2e9abadfcf5f [term 1 FOLLOWER]: Raft consensus is shut down! I0720 00:44:37.968711 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179): perf score=1.011213 I0720 00:44:37.976377 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:37.987720 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.019s user 0.015s sys 0.004s Metrics: {"bytes_written":18123,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9779,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28295,"cfile_init":5,"data dirs.queue_time_us":596,"data dirs.run_cpu_time_us":494,"data dirs.run_wall_time_us":885,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":536,"lbm_reads_lt_1ms":46,"lbm_write_time_us":477,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":524,"spinlock_wait_cycles":1920} I0720 00:44:37.988620 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 5058 bytes on disk I0720 00:44:37.989085 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4} I0720 00:44:37.989485 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:37.989933 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(bd37441ffc54486b90ba5230a81da533): perf score=0.000300 I0720 00:44:37.998186 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.008s user 0.008s sys 0.000s Metrics: {"bytes_written":6433,"cfile_init":1,"data dirs.queue_time_us":66,"data dirs.run_cpu_time_us":117,"data dirs.run_wall_time_us":524,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":394,"lbm_writes_lt_1ms":23,"rows_written":68} I0720 00:44:37.998920 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533): perf score=0.948911 I0720 00:44:38.010637 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:38.016847 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":17734,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9448,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27318,"cfile_init":6,"data dirs.queue_time_us":619,"data dirs.run_cpu_time_us":338,"data dirs.run_wall_time_us":739,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":562,"lbm_reads_lt_1ms":50,"lbm_write_time_us":423,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":506} I0720 00:44:38.017642 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533): 4933 bytes on disk I0720 00:44:38.018048 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(bd37441ffc54486b90ba5230a81da533) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4} I0720 00:44:38.047565 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:38.068019 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(9a5c9555b3fa4c92940250278686a179): perf score=0.000282 I0720 00:44:38.077618 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.009s user 0.007s sys 0.000s Metrics: {"bytes_written":6705,"cfile_init":1,"data dirs.queue_time_us":95,"data dirs.run_cpu_time_us":117,"data dirs.run_wall_time_us":572,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":466,"lbm_writes_lt_1ms":23,"rows_written":79,"spinlock_wait_cycles":1920} I0720 00:44:38.078510 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 878 bytes on disk I0720 00:44:38.078969 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4} I0720 00:44:38.079486 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179): perf score=1.010830 I0720 00:44:38.100150 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.020s user 0.018s sys 0.000s Metrics: {"bytes_written":18123,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9779,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28223,"cfile_init":5,"data dirs.queue_time_us":778,"data dirs.run_cpu_time_us":364,"data dirs.run_wall_time_us":772,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":591,"lbm_reads_lt_1ms":46,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":524} I0720 00:44:38.101001 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179): 5058 bytes on disk I0720 00:44:38.101424 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(9a5c9555b3fa4c92940250278686a179) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} I0720 00:44:38.116379 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:38.163581 7608 consensus_queue.cc:544] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:38.163744 7608 consensus_peers.cc:458] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.164260 7732 consensus_queue.cc:544] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:38.164381 7732 consensus_peers.cc:458] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.165971 7423 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 4d36dab0fb81459dbc2df72f66852ea2 (attempt 1) I0720 00:44:38.167652 7421 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet eb86205d3d8548f4b0daaa602f89c3a9 (attempt 1) I0720 00:44:38.168484 7672 consensus_queue.cc:206] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:38.169021 7794 consensus_queue.cc:206] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:38.169862 7798 raft_consensus.cc:1184] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:38.170361 7907 consensus_queue.cc:984] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:38.170492 7672 raft_consensus.cc:1184] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:38.170922 8052 consensus_queue.cc:984] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:38.172293 8200 raft_consensus.cc:2792] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.172758 7798 raft_consensus.cc:2792] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.173009 8052 raft_consensus.cc:2792] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.175565 7672 raft_consensus.cc:2792] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.176623 7410 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 4d36dab0fb81459dbc2df72f66852ea2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:38.177258 7421 catalog_manager.cc:4067] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:38.179098 7408 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet eb86205d3d8548f4b0daaa602f89c3a9 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:38.181812 7421 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 4d36dab0fb81459dbc2df72f66852ea2 (attempt 1) I0720 00:44:38.183006 7672 consensus_queue.cc:206] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:38.183360 7421 catalog_manager.cc:4067] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:38.184715 7798 raft_consensus.cc:1184] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:38.185205 8202 consensus_queue.cc:984] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:38.185891 7421 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet eb86205d3d8548f4b0daaa602f89c3a9 (attempt 1) I0720 00:44:38.187149 7794 consensus_queue.cc:206] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:38.188808 7907 raft_consensus.cc:2792] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.189432 7798 raft_consensus.cc:2792] T 4d36dab0fb81459dbc2df72f66852ea2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.191977 7410 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4d36dab0fb81459dbc2df72f66852ea2 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:38.193032 7672 raft_consensus.cc:1184] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:38.193483 8203 consensus_queue.cc:984] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:38.193624 7423 catalog_manager.cc:4067] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:38.195245 7608 consensus_queue.cc:544] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:38.195345 7608 consensus_peers.cc:458] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.195857 7670 raft_consensus.cc:2792] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.199908 7421 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 9a5c9555b3fa4c92940250278686a179 (attempt 1) I0720 00:44:38.200506 8052 raft_consensus.cc:2792] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.201074 7670 consensus_queue.cc:206] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } W0720 00:44:38.202028 7729 consensus_peers.cc:458] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb86205d3d8548f4b0daaa602f89c3a9. This is attempt 1: this message will repeat every 5th retry. W0720 00:44:38.202263 7605 consensus_peers.cc:458] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d36dab0fb81459dbc2df72f66852ea2. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.202608 7798 raft_consensus.cc:1184] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:38.203004 7907 consensus_queue.cc:984] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:38.202965 7421 catalog_manager.cc:4067] T eb86205d3d8548f4b0daaa602f89c3a9 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.204406 7408 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet eb86205d3d8548f4b0daaa602f89c3a9 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:38.204934 7907 raft_consensus.cc:2792] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.207585 7410 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 9a5c9555b3fa4c92940250278686a179 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:38.208178 7798 raft_consensus.cc:2792] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.208400 7423 catalog_manager.cc:4067] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index -1 to 105, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:38.210436 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9a5c9555b3fa4c92940250278686a179 (attempt 1) I0720 00:44:38.211793 7672 consensus_queue.cc:206] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:38.212340 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 9a5c9555b3fa4c92940250278686a179 (attempt 1) I0720 00:44:38.213094 7798 raft_consensus.cc:1184] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) W0720 00:44:38.213347 7605 consensus_peers.cc:458] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a5c9555b3fa4c92940250278686a179. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.213611 8202 consensus_queue.cc:984] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:38.213816 7410 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9a5c9555b3fa4c92940250278686a179 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } }. Pending config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:38.213960 7410 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9a5c9555b3fa4c92940250278686a179 with cas_config_opid_index 105 with a delay of 32 ms (attempt = 1) I0720 00:44:38.215265 7907 raft_consensus.cc:2792] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.215759 7798 raft_consensus.cc:2792] T 9a5c9555b3fa4c92940250278686a179 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.217975 7410 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9a5c9555b3fa4c92940250278686a179 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:38.218981 7423 catalog_manager.cc:4067] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index 105 to 106, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:38.246729 7410 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9a5c9555b3fa4c92940250278686a179 with cas_config_opid_index 105: aborting the task: latest config opid_index 106; task opid_index 105 I0720 00:44:38.249399 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:38.273555 7732 consensus_queue.cc:544] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:38.273689 7732 consensus_peers.cc:458] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.275545 7421 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet a487d770322e4226b00934877a3bff5d (attempt 1) I0720 00:44:38.276732 7798 consensus_queue.cc:206] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:38.278322 7672 raft_consensus.cc:1184] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I0720 00:44:38.278844 8203 consensus_queue.cc:984] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:38.281539 8201 raft_consensus.cc:2792] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.282042 7672 raft_consensus.cc:2792] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.284080 7408 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet a487d770322e4226b00934877a3bff5d with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:38.285399 7421 catalog_manager.cc:4067] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.288038 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a487d770322e4226b00934877a3bff5d (attempt 1) I0720 00:44:38.289129 7798 consensus_queue.cc:206] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } W0720 00:44:38.290994 7729 consensus_peers.cc:458] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a487d770322e4226b00934877a3bff5d. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.291160 7672 raft_consensus.cc:1184] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 107. (index mismatch) I0720 00:44:38.291708 8201 consensus_queue.cc:984] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:38.293356 8203 raft_consensus.cc:2792] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.293805 7672 raft_consensus.cc:2792] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.295663 7408 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a487d770322e4226b00934877a3bff5d with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:38.297019 7423 catalog_manager.cc:4067] T a487d770322e4226b00934877a3bff5d P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.316246 7608 consensus_queue.cc:544] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:38.316395 7608 consensus_peers.cc:458] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.318053 7421 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet bd37441ffc54486b90ba5230a81da533 (attempt 1) I0720 00:44:38.319149 7672 consensus_queue.cc:206] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:38.320559 7798 raft_consensus.cc:1184] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0720 00:44:38.321043 8202 consensus_queue.cc:984] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:38.322829 7907 raft_consensus.cc:2792] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.323349 7798 raft_consensus.cc:2792] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:38.325623 7410 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet bd37441ffc54486b90ba5230a81da533 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:38.326938 7421 catalog_manager.cc:4067] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index -1 to 105, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:38.329349 7421 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet bd37441ffc54486b90ba5230a81da533 (attempt 1) I0720 00:44:38.330627 7672 consensus_queue.cc:206] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:38.331483 7421 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet bd37441ffc54486b90ba5230a81da533 (attempt 1) I0720 00:44:38.332432 7798 raft_consensus.cc:1184] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) W0720 00:44:38.332615 7605 consensus_peers.cc:458] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd37441ffc54486b90ba5230a81da533. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:38.332866 7907 consensus_queue.cc:984] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:38.332957 7410 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd37441ffc54486b90ba5230a81da533 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } }. Pending config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:38.333037 7410 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd37441ffc54486b90ba5230a81da533 with cas_config_opid_index 105 with a delay of 63 ms (attempt = 1) I0720 00:44:38.334493 8202 raft_consensus.cc:2792] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.334941 7798 raft_consensus.cc:2792] T bd37441ffc54486b90ba5230a81da533 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 105 to 106, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:38.336608 7410 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd37441ffc54486b90ba5230a81da533 with cas_config_opid_index 105: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:38.337913 7421 catalog_manager.cc:4067] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index 105 to 106, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 1 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:38.352458 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.000291 I0720 00:44:38.360528 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: FlushMRSOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.008s user 0.002s sys 0.004s Metrics: {"bytes_written":5352,"cfile_init":1,"data dirs.queue_time_us":67,"data dirs.run_cpu_time_us":110,"data dirs.run_wall_time_us":531,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":430,"lbm_writes_lt_1ms":23,"rows_written":27} I0720 00:44:38.361488 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 384 bytes on disk I0720 00:44:38.361968 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4} I0720 00:44:38.362579 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5): perf score=0.889060 I0720 00:44:38.381227 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: CompactRowSetsOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.018s user 0.014s sys 0.004s Metrics: {"bytes_written":18105,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9757,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27973,"cfile_init":5,"data dirs.queue_time_us":497,"data dirs.run_cpu_time_us":618,"data dirs.run_wall_time_us":949,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":555,"lbm_reads_lt_1ms":46,"lbm_write_time_us":445,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"rows_written":523,"spinlock_wait_cycles":25600} I0720 00:44:38.382205 7718 maintenance_manager.cc:327] P 7a005754141d43ac91f33e8797a489ab: Scheduling UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5): 5042 bytes on disk I0720 00:44:38.382716 7612 maintenance_manager.cc:532] P 7a005754141d43ac91f33e8797a489ab: UndoDeltaBlockGCOp(7b95ae04955844828ac3ced5e85c9ed5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4} I0720 00:44:38.397006 7410 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bd37441ffc54486b90ba5230a81da533 with cas_config_opid_index 105: aborting the task: latest config opid_index 106; task opid_index 105 I0720 00:44:38.510417 234 disk_failure-itest.cc:256] Currently has 0 failed tablets I0720 00:44:38.519822 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(a487d770322e4226b00934877a3bff5d): perf score=0.000335 I0720 00:44:38.528283 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.008s user 0.006s sys 0.000s Metrics: {"bytes_written":5544,"cfile_init":1,"data dirs.queue_time_us":293,"data dirs.run_cpu_time_us":116,"data dirs.run_wall_time_us":546,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":448,"lbm_writes_lt_1ms":23,"rows_written":33,"thread_start_us":181,"threads_started":1} I0720 00:44:38.529202 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(a487d770322e4226b00934877a3bff5d): perf score=0.996076 I0720 00:44:38.548854 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.019s user 0.012s sys 0.007s Metrics: {"bytes_written":18359,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9925,"cfile_cache_miss":26,"cfile_cache_miss_bytes":28689,"cfile_init":6,"data dirs.queue_time_us":1455,"data dirs.run_cpu_time_us":493,"data dirs.run_wall_time_us":841,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":570,"lbm_reads_lt_1ms":50,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"mutex_wait_us":56,"num_input_rowsets":2,"rows_written":532,"spinlock_wait_cycles":4736,"thread_start_us":415,"threads_started":2} I0720 00:44:38.549805 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d): 5155 bytes on disk I0720 00:44:38.550341 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(a487d770322e4226b00934877a3bff5d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4} I0720 00:44:38.550927 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.000318 I0720 00:44:38.558382 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: FlushMRSOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.007s user 0.007s sys 0.000s Metrics: {"bytes_written":4908,"cfile_init":1,"data dirs.queue_time_us":59,"data dirs.run_cpu_time_us":93,"data dirs.run_wall_time_us":474,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":392,"lbm_writes_lt_1ms":23,"rows_written":10,"thread_start_us":227,"threads_started":1,"wal-append.queue_time_us":340} I0720 00:44:38.559249 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 224 bytes on disk I0720 00:44:38.559725 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4} I0720 00:44:38.560176 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b): perf score=0.782508 I0720 00:44:38.578203 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: CompactRowSetsOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.018s user 0.018s sys 0.000s Metrics: {"bytes_written":17493,"cfile_cache_hit":10,"cfile_cache_hit_bytes":9327,"cfile_cache_miss":26,"cfile_cache_miss_bytes":27045,"cfile_init":5,"data dirs.queue_time_us":964,"data dirs.run_cpu_time_us":445,"data dirs.run_wall_time_us":820,"delta_iterators_relevant":4,"drs_written":1,"lbm_read_time_us":531,"lbm_reads_lt_1ms":46,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"mutex_wait_us":37,"num_input_rowsets":2,"rows_written":499,"spinlock_wait_cycles":45824} I0720 00:44:38.579067 7842 maintenance_manager.cc:327] P c3780073b3594c88b678ec4da2bd8cda: Scheduling UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b): 4837 bytes on disk I0720 00:44:38.579556 7736 maintenance_manager.cc:532] P c3780073b3594c88b678ec4da2bd8cda: UndoDeltaBlockGCOp(0380f4644f22404e9f2b80565c2d7a0b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4} W0720 00:44:38.610286 8202 consensus_peers.cc:251] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:38.670115 8201 consensus_peers.cc:251] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:38.710124 8202 consensus_peers.cc:251] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:38.733064 8201 consensus_peers.cc:251] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:38.845947 8202 raft_consensus.cc:472] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:38.846112 8202 raft_consensus.cc:494] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:38.846691 8202 leader_election.cc:296] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:38.847141 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:38.847223 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:38.847363 7798 raft_consensus.cc:2311] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a005754141d43ac91f33e8797a489ab for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:38.847790 7608 leader_election.cc:348] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:38.847909 7606 leader_election.cc:310] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda I0720 00:44:38.848134 8202 raft_consensus.cc:2592] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:38.878165 8202 consensus_peers.cc:251] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:38.902834 8180 heartbeater.cc:446] Master 127.0.58.190:34075 was elected leader, sending a full tablet report... I0720 00:44:39.027251 234 disk_failure-itest.cc:256] Currently has 10 failed tablets I0720 00:44:39.027302 234 disk_failure-itest.cc:234] Resetting error injection flags W0720 00:44:39.028138 7589 generic_service.cc:128] {username='slave'} at 127.0.0.1:51578 forcing change of non-runtime-safe flag env_inject_eio I0720 00:44:39.028213 7589 generic_service.cc:148] {username='slave'} at 127.0.0.1:51578 changed flags via RPC: env_inject_eio from '1' to '0.0' W0720 00:44:39.029157 7589 generic_service.cc:128] {username='slave'} at 127.0.0.1:51578 forcing change of non-runtime-safe flag cfile_inject_corruption I0720 00:44:39.029219 7589 generic_service.cc:148] {username='slave'} at 127.0.0.1:51578 changed flags via RPC: cfile_inject_corruption from '0' to '0.0' W0720 00:44:39.029974 7589 generic_service.cc:128] {username='slave'} at 127.0.0.1:51578 forcing change of non-runtime-safe flag enable_tablet_copy I0720 00:44:39.030035 7589 generic_service.cc:148] {username='slave'} at 127.0.0.1:51578 changed flags via RPC: enable_tablet_copy from 'false' to 'true' W0720 00:44:39.035099 8202 consensus_peers.cc:251] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:39.094099 7758 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0d5f343fde1d404598e962001ef17ae2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1563583478093714 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1563583477930234 usec, L: 0 Physical time difference: 0.163s I0720 00:44:39.094722 7758 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:41274 (request call id 1) took 1003ms. Request Metrics: {} W0720 00:44:39.108654 8201 consensus_peers.cc:251] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:39.111286 7758 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0d5f343fde1d404598e962001ef17ae2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.180s secs, (max is 0.750s): W0720 00:44:39.163535 7627 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0d5f343fde1d404598e962001ef17ae2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.435s secs, (max is 0.750s): W0720 00:44:39.210438 7758 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0d5f343fde1d404598e962001ef17ae2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.279s secs, (max is 0.750s): W0720 00:44:39.219075 8202 consensus_peers.cc:251] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:39.229768 8202 raft_consensus.cc:472] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.229905 8202 raft_consensus.cc:494] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:39.230423 8202 leader_election.cc:296] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.230880 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:39.231096 7798 raft_consensus.cc:2311] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a005754141d43ac91f33e8797a489ab for term 2 because replica is either leader or believes a valid leader to be alive. I0720 00:44:39.231009 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true W0720 00:44:39.231549 7608 leader_election.cc:348] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.231633 7608 leader_election.cc:310] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda I0720 00:44:39.231916 8202 raft_consensus.cc:2592] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0720 00:44:39.237720 8201 raft_consensus.cc:472] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.237848 8201 raft_consensus.cc:494] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.238364 8201 leader_election.cc:296] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:39.238906 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 104 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:39.238929 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 104 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.239084 7672 raft_consensus.cc:2311] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3780073b3594c88b678ec4da2bd8cda for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:39.239409 7732 leader_election.cc:348] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.239580 7732 leader_election.cc:310] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab I0720 00:44:39.239809 8201 raft_consensus.cc:2592] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0720 00:44:39.248800 8202 raft_consensus.cc:472] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.248950 8202 raft_consensus.cc:494] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.249505 8202 leader_election.cc:296] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:39.249999 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.250008 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:39.250231 7798 raft_consensus.cc:2311] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a005754141d43ac91f33e8797a489ab for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:39.250530 7608 leader_election.cc:348] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.250671 7606 leader_election.cc:310] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, c3780073b3594c88b678ec4da2bd8cda I0720 00:44:39.250911 8202 raft_consensus.cc:2592] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:39.268048 8202 consensus_peers.cc:251] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:39.285333 8201 raft_consensus.cc:472] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.285531 8201 raft_consensus.cc:494] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.286190 8201 leader_election.cc:296] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487) I0720 00:44:39.286818 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.286888 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:39.287111 7672 raft_consensus.cc:2311] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3780073b3594c88b678ec4da2bd8cda for term 2 because replica is either leader or believes a valid leader to be alive. W0720 00:44:39.287364 7732 leader_election.cc:348] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.287534 7732 leader_election.cc:310] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f, 7a005754141d43ac91f33e8797a489ab I0720 00:44:39.287783 8201 raft_consensus.cc:2592] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0720 00:44:39.315258 8201 consensus_peers.cc:251] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:39.317479 7627 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0d5f343fde1d404598e962001ef17ae2" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.589s secs, (max is 0.750s): I0720 00:44:39.354404 8201 raft_consensus.cc:472] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.354549 8201 raft_consensus.cc:494] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.355116 8201 leader_election.cc:296] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.355731 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:39.355731 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.355957 7672 raft_consensus.cc:2350] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. W0720 00:44:39.356257 7732 leader_election.cc:348] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.356410 7732 leader_election.cc:310] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.356659 8201 raft_consensus.cc:2644] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:39.356726 8201 raft_consensus.cc:472] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.356779 8201 raft_consensus.cc:2890] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:39.358402 8201 raft_consensus.cc:494] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.358809 8201 leader_election.cc:296] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.359344 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:39.359333 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "c8db08710314422cbd1b616d57ec4cec" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:39.359496 7672 raft_consensus.cc:2890] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:39.359839 7732 leader_election.cc:348] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.361099 7672 raft_consensus.cc:2350] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 2. I0720 00:44:39.361433 7732 leader_election.cc:310] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.361718 8201 raft_consensus.cc:2644] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:39.362027 8201 raft_consensus.cc:667] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Becoming Leader. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Running, Role: LEADER I0720 00:44:39.362283 8201 consensus_queue.cc:206] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.367357 8202 raft_consensus.cc:472] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.367486 8202 raft_consensus.cc:494] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.368010 8202 leader_election.cc:296] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:39.367784 7423 catalog_manager.cc:4067] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda reported cstate change: term changed from 1 to 2, leader changed from 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) to c3780073b3594c88b678ec4da2bd8cda (127.0.58.131). New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.368511 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true W0720 00:44:39.369105 7608 leader_election.cc:348] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.369300 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:39.369493 7798 raft_consensus.cc:2350] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. I0720 00:44:39.369804 7606 leader_election.cc:310] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.370066 8202 raft_consensus.cc:2644] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:39.370170 8202 raft_consensus.cc:472] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.370221 8202 raft_consensus.cc:2890] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:39.371760 8202 raft_consensus.cc:494] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.372524 8202 leader_election.cc:296] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:39.372543 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:39.372623 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "5570d23334524c9a9e1bbbdf1c4c3806" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:39.372726 7798 raft_consensus.cc:2890] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:39.373122 7608 leader_election.cc:348] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.374826 7798 raft_consensus.cc:2350] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 2. I0720 00:44:39.375151 7606 leader_election.cc:310] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.375427 8202 raft_consensus.cc:2644] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:39.375946 8202 raft_consensus.cc:667] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 2 LEADER]: Becoming Leader. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Running, Role: LEADER I0720 00:44:39.376140 8202 consensus_queue.cc:206] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.378206 7423 catalog_manager.cc:4067] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab reported cstate change: term changed from 1 to 2, leader changed from 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) to 7a005754141d43ac91f33e8797a489ab (127.0.58.130). New cstate: current_term: 2 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } I0720 00:44:39.432190 8201 raft_consensus.cc:472] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.432358 8201 raft_consensus.cc:494] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:39.432937 8201 leader_election.cc:296] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.433466 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:39.433534 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.433676 7672 raft_consensus.cc:2350] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. W0720 00:44:39.434089 7732 leader_election.cc:348] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.434235 7732 leader_election.cc:310] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.434481 8201 raft_consensus.cc:2644] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:39.434537 8201 raft_consensus.cc:472] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.434590 8201 raft_consensus.cc:2890] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:39.435679 8219 raft_consensus.cc:472] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.435809 8219 raft_consensus.cc:494] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.436262 8219 leader_election.cc:296] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.436209 8201 raft_consensus.cc:494] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:39.436734 8201 leader_election.cc:296] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.436770 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" is_pre_election: true I0720 00:44:39.436802 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.436969 7672 raft_consensus.cc:2350] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 1. I0720 00:44:39.437252 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" W0720 00:44:39.437309 7732 leader_election.cc:348] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.437366 7672 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0d5f343fde1d404598e962001ef17ae2" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:39.437448 7732 leader_election.cc:310] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.437551 7672 raft_consensus.cc:2890] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:39.437753 8201 raft_consensus.cc:2644] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:39.437842 8201 raft_consensus.cc:472] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) W0720 00:44:39.437827 7732 leader_election.cc:348] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.437909 8201 raft_consensus.cc:2890] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:39.438155 8202 consensus_peers.cc:251] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:39.439491 8201 raft_consensus.cc:494] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.439949 8201 leader_election.cc:296] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Requested vote from peers 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487), 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263) I0720 00:44:39.440347 7672 raft_consensus.cc:2350] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 2. I0720 00:44:39.440448 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:39.440582 7670 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" candidate_uuid: "c3780073b3594c88b678ec4da2bd8cda" candidate_term: 2 candidate_status { last_received { term: 1 index: 105 } } ignore_live_leader: false dest_uuid: "7a005754141d43ac91f33e8797a489ab" I0720 00:44:39.440712 7670 raft_consensus.cc:2890] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:39.442057 7670 raft_consensus.cc:2350] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3780073b3594c88b678ec4da2bd8cda in term 2. W0720 00:44:39.442355 7732 leader_election.cc:348] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.442515 7732 leader_election.cc:310] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.442766 8201 raft_consensus.cc:2644] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:39.442826 7732 leader_election.cc:310] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.442886 8201 raft_consensus.cc:667] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Becoming Leader. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Running, Role: LEADER I0720 00:44:39.443043 8219 raft_consensus.cc:2644] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:39.443426 8201 consensus_queue.cc:206] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } I0720 00:44:39.443663 8219 raft_consensus.cc:667] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Becoming Leader. State: Replica: c3780073b3594c88b678ec4da2bd8cda, State: Running, Role: LEADER I0720 00:44:39.443837 8219 consensus_queue.cc:206] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.445749 7423 catalog_manager.cc:4067] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: term changed from 1 to 2, leader changed from 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) to c3780073b3594c88b678ec4da2bd8cda (127.0.58.131). New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } } I0720 00:44:39.448413 8202 raft_consensus.cc:472] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.448520 8202 raft_consensus.cc:494] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.449057 8202 leader_election.cc:296] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:39.449452 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 104 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" is_pre_election: true I0720 00:44:39.449481 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 104 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" is_pre_election: true I0720 00:44:39.449615 7798 raft_consensus.cc:2350] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 1. W0720 00:44:39.449915 7608 leader_election.cc:348] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.450060 7606 leader_election.cc:310] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.450016 7423 catalog_manager.cc:4067] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda reported cstate change: term changed from 1 to 2, leader changed from 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) to c3780073b3594c88b678ec4da2bd8cda (127.0.58.131). New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.450346 8202 raft_consensus.cc:2644] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Leader pre-election won for term 2 I0720 00:44:39.450425 8202 raft_consensus.cc:472] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 110cfca4c9df4d82bf4a2e9abadfcf5f) I0720 00:44:39.450480 8202 raft_consensus.cc:2890] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 1 FOLLOWER]: Advancing to term 2 I0720 00:44:39.452265 8202 raft_consensus.cc:494] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.452828 8202 leader_election.cc:296] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 election: Requested vote from peers 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263), c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) I0720 00:44:39.453212 7545 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 104 } } ignore_live_leader: false dest_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" I0720 00:44:39.453222 7798 tablet_service.cc:1295] Received RequestConsensusVote() RPC: tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" candidate_uuid: "7a005754141d43ac91f33e8797a489ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 104 } } ignore_live_leader: false dest_uuid: "c3780073b3594c88b678ec4da2bd8cda" I0720 00:44:39.453349 7798 raft_consensus.cc:2890] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 1 FOLLOWER]: Advancing to term 2 W0720 00:44:39.453626 7608 leader_election.cc:348] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Illegal state: cannot vote while shut down I0720 00:44:39.454846 7798 raft_consensus.cc:2350] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a005754141d43ac91f33e8797a489ab in term 2. I0720 00:44:39.455152 7606 leader_election.cc:310] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7a005754141d43ac91f33e8797a489ab, c3780073b3594c88b678ec4da2bd8cda; no voters: 110cfca4c9df4d82bf4a2e9abadfcf5f I0720 00:44:39.455425 8202 raft_consensus.cc:2644] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Leader election won for term 2 I0720 00:44:39.455529 8202 raft_consensus.cc:667] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 2 LEADER]: Becoming Leader. State: Replica: 7a005754141d43ac91f33e8797a489ab, State: Running, Role: LEADER I0720 00:44:39.455713 8202 consensus_queue.cc:206] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 104, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.457478 7423 catalog_manager.cc:4067] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab reported cstate change: term changed from 1 to 2, leader changed from 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) to 7a005754141d43ac91f33e8797a489ab (127.0.58.130). New cstate: current_term: 2 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110cfca4c9df4d82bf4a2e9abadfcf5f" member_type: VOTER last_known_addr { host: "127.0.58.129" port: 42263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: UNKNOWN } } } W0720 00:44:39.596124 8219 consensus_peers.cc:251] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:39.683249 8202 consensus_peers.cc:251] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:39.738968 7627 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "c8db08710314422cbd1b616d57ec4cec" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437932016381952 propagated_timestamp: 6404437933002760192 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 1.991s secs, (max is 0.750s): W0720 00:44:39.739850 8202 consensus_peers.cc:251] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:39.762740 7627 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "c8db08710314422cbd1b616d57ec4cec" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437932016381952 propagated_timestamp: 6404437933002760192 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 2.015s secs, (max is 0.750s): I0720 00:44:39.778513 7798 raft_consensus.cc:1184] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:39.779011 8202 consensus_queue.cc:984] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:39.783704 7608 consensus_queue.cc:544] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:39.783823 7608 consensus_peers.cc:458] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.784873 7423 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 5570d23334524c9a9e1bbbdf1c4c3806 (attempt 1) I0720 00:44:39.785989 7670 consensus_queue.cc:206] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.787415 7798 raft_consensus.cc:1184] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:39.787842 8225 consensus_queue.cc:984] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:39.789630 8234 raft_consensus.cc:2792] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.790136 7798 raft_consensus.cc:2792] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.792150 7410 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 5570d23334524c9a9e1bbbdf1c4c3806 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:39.792987 7423 catalog_manager.cc:4067] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.795433 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5570d23334524c9a9e1bbbdf1c4c3806 (attempt 1) I0720 00:44:39.796506 7670 consensus_queue.cc:206] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:39.797919 7798 raft_consensus.cc:1184] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) W0720 00:44:39.798199 7605 consensus_peers.cc:458] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5570d23334524c9a9e1bbbdf1c4c3806. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.798427 8234 consensus_queue.cc:984] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:39.799998 8225 raft_consensus.cc:2792] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.800492 7798 raft_consensus.cc:2792] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.802776 7410 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5570d23334524c9a9e1bbbdf1c4c3806 with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:39.804143 7423 catalog_manager.cc:4067] T 5570d23334524c9a9e1bbbdf1c4c3806 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } W0720 00:44:39.826393 7627 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437932016381952 propagated_timestamp: 6404437933360115712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 2.481s secs, (max is 0.750s): I0720 00:44:39.829686 7670 raft_consensus.cc:1184] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:39.830134 8219 consensus_queue.cc:984] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:39.833441 7732 consensus_queue.cc:544] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:39.833546 7732 consensus_peers.cc:458] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.835803 7423 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 0d5f343fde1d404598e962001ef17ae2 (attempt 1) I0720 00:44:39.836757 7798 consensus_queue.cc:206] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } I0720 00:44:39.837994 7670 raft_consensus.cc:1184] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:39.838470 8227 consensus_queue.cc:984] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:39.840219 8201 raft_consensus.cc:2792] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } } I0720 00:44:39.840729 7670 raft_consensus.cc:2792] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } } W0720 00:44:39.842042 7627 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "0380f4644f22404e9f2b80565c2d7a0b" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437932016381952 propagated_timestamp: 6404437933360115712 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 2.496s secs, (max is 0.750s): I0720 00:44:39.842667 7408 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 0d5f343fde1d404598e962001ef17ae2 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:39.844085 7423 catalog_manager.cc:4067] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.846565 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 0d5f343fde1d404598e962001ef17ae2 (attempt 1) I0720 00:44:39.847659 7798 consensus_queue.cc:206] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:39.849020 7670 raft_consensus.cc:1184] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) W0720 00:44:39.849300 7729 consensus_peers.cc:458] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d5f343fde1d404598e962001ef17ae2. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.849426 8227 consensus_queue.cc:984] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:39.851006 8219 raft_consensus.cc:2792] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.851496 7670 raft_consensus.cc:2792] T 0d5f343fde1d404598e962001ef17ae2 P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.853310 7408 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0d5f343fde1d404598e962001ef17ae2 with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:39.854193 7423 catalog_manager.cc:4067] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:39.862159 7670 raft_consensus.cc:1184] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:39.862583 8219 consensus_queue.cc:984] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:39.866230 7732 consensus_queue.cc:544] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:39.866470 7732 consensus_peers.cc:458] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.867622 7423 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet c8db08710314422cbd1b616d57ec4cec (attempt 1) I0720 00:44:39.868582 7798 consensus_queue.cc:206] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.869724 7670 raft_consensus.cc:1184] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:39.870234 8219 consensus_queue.cc:984] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:39.871937 8201 raft_consensus.cc:2792] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.872475 7670 raft_consensus.cc:2792] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.874678 7408 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet c8db08710314422cbd1b616d57ec4cec with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:39.875932 7423 catalog_manager.cc:4067] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.878605 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c8db08710314422cbd1b616d57ec4cec (attempt 1) I0720 00:44:39.879806 7798 consensus_queue.cc:206] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:39.880679 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c8db08710314422cbd1b616d57ec4cec (attempt 1) W0720 00:44:39.881633 7729 consensus_peers.cc:458] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8db08710314422cbd1b616d57ec4cec. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.882153 7670 raft_consensus.cc:1184] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) I0720 00:44:39.882616 7408 catalog_manager.cc:3618] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c8db08710314422cbd1b616d57ec4cec with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER failed with leader c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } }. Pending config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:39.882716 7408 catalog_manager.cc:3234] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c8db08710314422cbd1b616d57ec4cec with cas_config_opid_index 107 with a delay of 64 ms (attempt = 1) I0720 00:44:39.882675 8219 consensus_queue.cc:984] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s W0720 00:44:39.883704 7758 tablet_service.cc:2419] Error setting up scanner with request new_scan_request { tablet_id: "7b95ae04955844828ac3ced5e85c9ed5" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true } read_mode: READ_AT_SNAPSHOT snap_timestamp: 6404437932016381952 propagated_timestamp: 6404437933594398720 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\241\312\311\351\005\"3\n\005slave\022*\n db2f82abcef04875b1b1ca569ecb5d7e\020\001\030\001 \001(\001" signature: "ZzS.,\355\374\366.\"\322\030\301[p\023F\257\210\375\013R\204\300\375\031\261*\373\357\177\256\323?tf\350\302\n:\027,\212\214X\034\0139\204\317\024\232,n:]\266?\226\242A6\203\243" signing_key_seq_num: 0 } } call_seq_id: 0: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Tablet hasn't heard from leader, or there hasn't been a stable leader for: 2.146s secs, (max is 0.750s): I0720 00:44:39.884553 8201 raft_consensus.cc:2792] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.885093 7670 raft_consensus.cc:2792] T c8db08710314422cbd1b616d57ec4cec P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.887459 7408 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c8db08710314422cbd1b616d57ec4cec with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:39.888386 7423 catalog_manager.cc:4067] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:39.932538 7798 raft_consensus.cc:1184] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 2 index: 105. (index mismatch) I0720 00:44:39.933105 8225 consensus_queue.cc:984] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0720 00:44:39.937482 7608 consensus_queue.cc:544] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:39.937577 7608 consensus_peers.cc:458] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.938489 7732 consensus_queue.cc:544] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: The tablet replica hosted on peer 110cfca4c9df4d82bf4a2e9abadfcf5f has failed W0720 00:44:39.938616 7732 consensus_peers.cc:458] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda -> Peer 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): Couldn't send request to peer 110cfca4c9df4d82bf4a2e9abadfcf5f. Error code: TABLET_FAILED (20). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: failing tablet. This is attempt 1: this message will repeat every 5th retry. W0720 00:44:39.940011 8219 consensus_peers.cc:251] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:39.941121 7421 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 7b95ae04955844828ac3ced5e85c9ed5 (attempt 1) I0720 00:44:39.941161 7670 raft_consensus.cc:1184] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:39.941952 8219 consensus_queue.cc:984] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:39.942574 7670 consensus_queue.cc:206] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 2.105, Last appended by leader: 104, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.944617 7798 raft_consensus.cc:1184] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch) I0720 00:44:39.945583 8225 consensus_queue.cc:984] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I0720 00:44:39.946367 7421 catalog_manager.cc:3818] Sending ChangeConfig:REMOVE_PEER on tablet 0380f4644f22404e9f2b80565c2d7a0b (attempt 1) I0720 00:44:39.947422 7794 consensus_queue.cc:206] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } I0720 00:44:39.947813 7408 catalog_manager.cc:3634] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c8db08710314422cbd1b616d57ec4cec with cas_config_opid_index 107: aborting the task: latest config opid_index 108; task opid_index 107 I0720 00:44:39.949013 7670 raft_consensus.cc:1184] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) I0720 00:44:39.949206 8225 raft_consensus.cc:2792] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 2 LEADER]: Committing config change with OpId 2.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.949477 8201 consensus_queue.cc:984] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:39.952193 8219 raft_consensus.cc:2792] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.952891 7798 raft_consensus.cc:2792] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Committing config change with OpId 2.106: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.953152 7410 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 7b95ae04955844828ac3ced5e85c9ed5 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:39.953433 7670 raft_consensus.cc:2792] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } } I0720 00:44:39.953948 7421 catalog_manager.cc:4067] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab reported cstate change: config changed from index -1 to 106, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.956084 7408 catalog_manager.cc:3601] ChangeConfig:REMOVE_PEER RPC for tablet 0380f4644f22404e9f2b80565c2d7a0b with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0720 00:44:39.957520 7421 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 7b95ae04955844828ac3ced5e85c9ed5 (attempt 1) I0720 00:44:39.957727 7423 catalog_manager.cc:4067] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index -1 to 107, VOTER 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129) evicted. New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } } I0720 00:44:39.958729 7672 consensus_queue.cc:206] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 2.106, Last appended by leader: 104, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } I0720 00:44:39.960232 7423 catalog_manager.cc:3686] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 0380f4644f22404e9f2b80565c2d7a0b (attempt 1) I0720 00:44:39.961319 7798 consensus_queue.cc:206] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 107, Committed index: 107, Last appended: 2.107, Last appended by leader: 105, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } W0720 00:44:39.962231 7605 consensus_peers.cc:458] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b95ae04955844828ac3ced5e85c9ed5. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.963070 7794 raft_consensus.cc:1184] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Refusing update from remote peer 7a005754141d43ac91f33e8797a489ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 106. Preceding OpId from leader: term: 2 index: 107. (index mismatch) W0720 00:44:39.963582 7729 consensus_peers.cc:458] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Couldn't send request to peer f828308d73154e0ebda8ef74a1ce2e08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0380f4644f22404e9f2b80565c2d7a0b. This is attempt 1: this message will repeat every 5th retry. I0720 00:44:39.963676 8225 consensus_queue.cc:984] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s I0720 00:44:39.964567 7672 raft_consensus.cc:1184] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Refusing update from remote peer c3780073b3594c88b678ec4da2bd8cda: Log matching property violated. Preceding OpId in replica: term: 2 index: 107. Preceding OpId from leader: term: 2 index: 108. (index mismatch) I0720 00:44:39.965791 8219 consensus_queue.cc:984] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 107, Time since last communication: 0.000s I0720 00:44:39.968612 8227 raft_consensus.cc:2792] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda [term 2 LEADER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.972532 7408 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0380f4644f22404e9f2b80565c2d7a0b with cas_config_opid_index 107: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:39.972688 8202 raft_consensus.cc:2792] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab [term 2 LEADER]: Committing config change with OpId 2.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.976518 7410 catalog_manager.cc:3601] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7b95ae04955844828ac3ced5e85c9ed5 with cas_config_opid_index 106: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0720 00:44:39.978582 7423 catalog_manager.cc:4067] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "c3780073b3594c88b678ec4da2bd8cda" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0720 00:44:39.980589 7672 raft_consensus.cc:2792] T 0380f4644f22404e9f2b80565c2d7a0b P 7a005754141d43ac91f33e8797a489ab [term 2 FOLLOWER]: Committing config change with OpId 2.108: config changed from index 107 to 108, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.984033 7798 raft_consensus.cc:2792] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda [term 2 FOLLOWER]: Committing config change with OpId 2.107: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } I0720 00:44:39.988495 7421 catalog_manager.cc:4067] T 7b95ae04955844828ac3ced5e85c9ed5 P c3780073b3594c88b678ec4da2bd8cda reported cstate change: config changed from index 106 to 107, NON_VOTER f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132) added. New cstate: current_term: 2 leader_uuid: "7a005754141d43ac91f33e8797a489ab" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "7a005754141d43ac91f33e8797a489ab" member_type: VOTER last_known_addr { host: "127.0.58.130" port: 40487 } } peers { permanent_uuid: "c3780073b3594c88b678ec4da2bd8cda" member_type: VOTER last_known_addr { host: "127.0.58.131" port: 37319 } } peers { permanent_uuid: "f828308d73154e0ebda8ef74a1ce2e08" member_type: NON_VOTER last_known_addr { host: "127.0.58.132" port: 36535 } attrs { promote: true } } } Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 4145b8157484456995b94b60104ebd7e | 127.0.58.190:34075 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 110cfca4c9df4d82bf4a2e9abadfcf5f | 127.0.58.129:42263 | HEALTHY | 7a005754141d43ac91f33e8797a489ab | 127.0.58.130:40487 | HEALTHY | c3780073b3594c88b678ec4da2bd8cda | 127.0.58.131:37319 | HEALTHY | f828308d73154e0ebda8ef74a1ce2e08 | 127.0.58.132:36535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:42263 enable_tablet_copy | false | unsafe | 127.0.58.130:40487, 127.0.58.131:37319, 127.0.58.132:36535 enable_tablet_copy | true | unsafe | 127.0.58.129:42263 env_inject_eio | 0 | hidden | 127.0.58.129:42263 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:42263 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:42263 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:40487 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:37319 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:36535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:42263 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:40487 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:37319 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:36535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Tablet 0380f4644f22404e9f2b80565c2d7a0b of table 'test-workload' is under-replicated: 1 replica(s) not RUNNING 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487): RUNNING 110cfca4c9df4d82bf4a2e9abadfcf5f (127.0.58.129:42263): not running State: FAILED Data state: TABLET_DATA_READY Last status: IO error: failing tablet c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319): RUNNING [LEADER] All reported replicas are: A = 7a005754141d43ac91f33e8797a489ab B = 110cfca4c9df4d82bf4a2e9abadfcf5f C = c3780073b3594c88b678ec4da2bd8cda D = f828308d73154e0ebda8ef74a1ce2e08 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+------------------------+--------------+--------------+------------ master | A B C* | | | Yes A | A C* D~ | 2 | 108 | Yes B | [config not available] | | | C | A C* D~ | 2 | 108 | Yes Tablet 7b95ae04955844828ac3ced5e85c9ed5 of table 'test-workload' is conflicted: 3 replicas' active configs disagree with the leader master's 7a005754141d43ac91f33e8797a489ab (127.0.58.130:40487): RUNNING [LEADER] c3780073b3594c88b678ec4da2bd8cda (127.0.58.131:37319): RUNNING All reported replicas are: A = 7a005754141d43ac91f33e8797a489ab B = c3780073b3594c88b678ec4da2bd8cda C = f828308d73154e0ebda8ef74a1ce2e08 The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A* B | | | Yes A | A* B C~ | 2 | 107 | Yes B | A* B C~ | 2 | 107 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+--------------------+---------------+---------+------------+------------------+------------- test-workload | 3 | CONSENSUS_MISMATCH | 10 | 8 | 0 | 1 | 1 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 1 First Quartile | 8 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 110cfca4c9df4d82bf4a2e9abadfcf5f | 127.0.58.129:42263 | 1 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 29 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0720 00:44:40.015532 234 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors W0720 00:44:40.020865 8202 consensus_peers.cc:251] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.154798 8202 consensus_peers.cc:251] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.199884 8219 consensus_peers.cc:251] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled Master Summary UUID | Address | Status ----------------------------------+--------------------+--------- 4145b8157484456995b94b60104ebd7e | 127.0.58.190:34075 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked master_tombstone_evicted_tablet_replicas | false | hidden | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/master-0/data-0/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+--------------------+---------+---------- 110cfca4c9df4d82bf4a2e9abadfcf5f | 127.0.58.129:42263 | HEALTHY | 7a005754141d43ac91f33e8797a489ab | 127.0.58.130:40487 | HEALTHY | c3780073b3594c88b678ec4da2bd8cda | 127.0.58.131:37319 | HEALTHY | f828308d73154e0ebda8ef74a1ce2e08 | 127.0.58.132:36535 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------+------------------------------------------------------------ cfile_inject_corruption | 0 | hidden | 127.0.58.129:42263 enable_tablet_copy | false | unsafe | 127.0.58.130:40487, 127.0.58.131:37319, 127.0.58.132:36535 enable_tablet_copy | true | unsafe | 127.0.58.129:42263 env_inject_eio | 0 | hidden | 127.0.58.129:42263 env_inject_eio_globs | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-1/** | hidden | 127.0.58.129:42263 flush_threshold_mb | 1 | runtime,experimental | all 4 server(s) checked flush_threshold_secs | 1 | runtime,experimental | all 4 server(s) checked ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.0.58.129 | experimental | 127.0.58.129:42263 local_ip_for_outbound_sockets | 127.0.58.130 | experimental | 127.0.58.130:40487 local_ip_for_outbound_sockets | 127.0.58.131 | experimental | 127.0.58.131:37319 local_ip_for_outbound_sockets | 127.0.58.132 | experimental | 127.0.58.132:36535 never_fsync | true | unsafe,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-0/data-0/info.pb | hidden | 127.0.58.129:42263 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-1/data-0/info.pb | hidden | 127.0.58.130:40487 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-2/data-0/info.pb | hidden | 127.0.58.131:37319 server_dump_info_path | /tmp/dist-test-taskAY5sm0/test-tmp/disk_failure-itest.0.DiskError_DiskErrorITest.TestFailDuringScanWorkload_1.1563583347463414-234/minicluster-data/ts-3/data-0/info.pb | hidden | 127.0.58.132:36535 Version Summary Version | Servers ------------------------+------------------------- 1.11.0-cdh6.x-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ---------------+----+---------+---------------+---------+------------+------------------+------------- test-workload | 3 | HEALTHY | 10 | 10 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Tablet Replica Count Outliers Type | UUID | Host | Replica Count -------+----------------------------------+--------------------+--------------- Small | 110cfca4c9df4d82bf4a2e9abadfcf5f | 127.0.58.129:42263 | 0 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 1 Tablets | 10 Replicas | 30 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0720 00:44:40.232225 234 log_verifier.cc:121] Checking tablet 0380f4644f22404e9f2b80565c2d7a0b W0720 00:44:40.242494 8202 consensus_peers.cc:251] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.256243 8202 consensus_peers.cc:251] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.284503 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet 0380f4644f22404e9f2b80565c2d7a0b I0720 00:44:40.284658 234 log_verifier.cc:121] Checking tablet 0d5f343fde1d404598e962001ef17ae2 W0720 00:44:40.335119 8219 consensus_peers.cc:251] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.338618 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet 0d5f343fde1d404598e962001ef17ae2 I0720 00:44:40.338765 234 log_verifier.cc:121] Checking tablet 4d36dab0fb81459dbc2df72f66852ea2 W0720 00:44:40.370088 8219 consensus_peers.cc:251] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.381795 8219 consensus_peers.cc:251] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.406240 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 4d36dab0fb81459dbc2df72f66852ea2 I0720 00:44:40.406388 234 log_verifier.cc:121] Checking tablet 5570d23334524c9a9e1bbbdf1c4c3806 W0720 00:44:40.458799 8219 consensus_peers.cc:251] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.459079 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet 5570d23334524c9a9e1bbbdf1c4c3806 I0720 00:44:40.459228 234 log_verifier.cc:121] Checking tablet 7b95ae04955844828ac3ced5e85c9ed5 I0720 00:44:40.512642 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet 7b95ae04955844828ac3ced5e85c9ed5 I0720 00:44:40.512789 234 log_verifier.cc:121] Checking tablet 9a5c9555b3fa4c92940250278686a179 W0720 00:44:40.525362 8202 consensus_peers.cc:251] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.556120 8202 consensus_peers.cc:251] T 4d36dab0fb81459dbc2df72f66852ea2 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.589282 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet 9a5c9555b3fa4c92940250278686a179 I0720 00:44:40.589416 234 log_verifier.cc:121] Checking tablet a487d770322e4226b00934877a3bff5d W0720 00:44:40.598184 8202 consensus_peers.cc:251] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.638805 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet a487d770322e4226b00934877a3bff5d I0720 00:44:40.638939 234 log_verifier.cc:121] Checking tablet bd37441ffc54486b90ba5230a81da533 W0720 00:44:40.678342 8202 consensus_peers.cc:251] T bd37441ffc54486b90ba5230a81da533 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.687849 8219 consensus_peers.cc:251] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.693192 234 log_verifier.cc:172] Verified matching terms for 106 ops in tablet bd37441ffc54486b90ba5230a81da533 I0720 00:44:40.693323 234 log_verifier.cc:121] Checking tablet c8db08710314422cbd1b616d57ec4cec W0720 00:44:40.713843 8219 consensus_peers.cc:251] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.746614 234 log_verifier.cc:172] Verified matching terms for 108 ops in tablet c8db08710314422cbd1b616d57ec4cec I0720 00:44:40.746764 234 log_verifier.cc:121] Checking tablet eb86205d3d8548f4b0daaa602f89c3a9 W0720 00:44:40.796272 8219 consensus_peers.cc:251] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.801884 234 log_verifier.cc:172] Verified matching terms for 107 ops in tablet eb86205d3d8548f4b0daaa602f89c3a9 I0720 00:44:40.802556 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 7460 I0720 00:44:40.803835 7589 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:40.818331 8202 consensus_peers.cc:251] T 5570d23334524c9a9e1bbbdf1c4c3806 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.912422 8219 consensus_peers.cc:251] T c8db08710314422cbd1b616d57ec4cec P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.915227 8202 consensus_peers.cc:251] T 7b95ae04955844828ac3ced5e85c9ed5 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:40.927973 8219 consensus_peers.cc:251] T eb86205d3d8548f4b0daaa602f89c3a9 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:40.960131 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 7460 I0720 00:44:40.986392 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 7596 I0720 00:44:40.987139 7714 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:41.137332 8202 consensus_peers.cc:251] T 9a5c9555b3fa4c92940250278686a179 P 7a005754141d43ac91f33e8797a489ab -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:41.138573 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 7596 W0720 00:44:41.161561 8219 consensus_peers.cc:251] T 0d5f343fde1d404598e962001ef17ae2 P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:41.163556 7407 connection.cc:655] server connection from 127.0.58.130:59015 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0720 00:44:41.163687 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 7720 I0720 00:44:41.164909 7838 generic_service.cc:171] Checking for leaks (request via RPC) W0720 00:44:41.311828 8219 consensus_peers.cc:251] T a487d770322e4226b00934877a3bff5d P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled W0720 00:44:41.311933 8293 consensus_peers.cc:251] T 0380f4644f22404e9f2b80565c2d7a0b P c3780073b3594c88b678ec4da2bd8cda -> Peer f828308d73154e0ebda8ef74a1ce2e08 (127.0.58.132:36535): Unable to generate Tablet Copy request for peer: Not implemented: Tablet Copy is disabled I0720 00:44:41.312510 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 7720 I0720 00:44:41.336241 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver pid 8059 I0720 00:44:41.337131 8177 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:41.380686 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-tserver with pid 8059 I0720 00:44:41.389694 234 external_mini_cluster.cc:1177] Attempting to check leaks for /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master pid 7398 I0720 00:44:41.390614 7456 generic_service.cc:171] Checking for leaks (request via RPC) I0720 00:44:41.473623 234 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskAY5sm0/build/asan/bin/kudu-master with pid 7398 [ OK ] DiskError/DiskErrorITest.TestFailDuringScanWorkload/1 (10368 ms) [----------] 4 tests from DiskError/DiskErrorITest (44940 ms total) [----------] Global test environment tear-down [==========] 8 tests from 2 test cases ran. (134003 ms total) [ PASSED ] 7 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DiskFailure/DiskFailureITest.TestFailDuringServerStartup/2, where GetParam() = ("log", false) 1 FAILED TEST