[==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from LinkedListTest [ RUN ] LinkedListTest.TestLoadAndVerify WARNING: Logging before InitGoogleLogging() is written to STDERR I0107 12:32:44.268944 24384 test_util.cc:229] Using random seed: -1381782132 I0107 12:32:44.272783 24384 linked_list-test.cc:100] Linked List Test Configuration: I0107 12:32:44.272794 24384 linked_list-test.cc:101] -------------- I0107 12:32:44.272797 24384 linked_list-test.cc:102] 50 chains I0107 12:32:44.272800 24384 linked_list-test.cc:103] 3 tablets I0107 12:32:44.272802 24384 linked_list-test.cc:104] Mutations on I0107 12:32:44.272804 24384 linked_list-test.cc:105] -------------- I0107 12:32:44.272823 24384 ts_itest-base.cc:112] Starting cluster with: I0107 12:32:44.272826 24384 ts_itest-base.cc:113] -------------- I0107 12:32:44.272828 24384 ts_itest-base.cc:114] 3 tablet servers I0107 12:32:44.272831 24384 ts_itest-base.cc:115] 3 replicas per TS I0107 12:32:44.272833 24384 ts_itest-base.cc:116] -------------- 2020-01-07T20:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-01-07T20:32:44Z Disabled control of system clock Could not open connection to daemon I0107 12:32:44.277325 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.208.62:54889 --webserver_interface=127.23.208.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:32:44.287662 24394 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:32:44.287760 24394 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:32:44.288694 24394 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0107 12:32:44.288709 24394 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:32:44.288715 24394 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0107 12:32:44.288720 24394 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0107 12:32:44.289911 24394 master_main.cc:96] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.208.62:54889 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/info.pb --webserver_interface=127.23.208.62 --webserver_port=0 --tablet_history_max_age_sec=86400 --never_fsync=true --heap_profile_path=/tmp/kudu-master.24394 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:32:44.291919 24394 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:32:44.293103 24394 hybrid_clock.cc:186] HybridClock initialized: now 1578429164293095 us; error 23 us; skew 500 ppm I0107 12:32:44.293180 24394 fs_manager.cc:263] Metadata directory not provided I0107 12:32:44.293201 24394 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:32:44.293252 24394 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0107 12:32:44.293263 24394 server_base.cc:453] Attempting to create new FS layout instead I0107 12:32:44.293898 24394 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/instance: uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.294030 24394 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/wal/instance: uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.294850 24394 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/data/block_manager_instance: No such file or directory (error 2) I0107 12:32:44.297214 24394 fs_manager.cc:518] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s I0107 12:32:44.297346 24394 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:32:44.297408 24394 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:32:44.298316 24394 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0107 12:32:44.298341 24394 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/wal uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.298368 24394 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:32:44.350270 24394 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:32:44.352943 24394 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.62:54889 I0107 12:32:44.353143 24394 webserver.cc:382] Webserver started at http://127.23.208.62:40214/ using document root and password file I0107 12:32:44.353261 24394 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/master-0/data/info.pb I0107 12:32:44.353504 24455 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.355191 24455 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e: Bootstrap starting. I0107 12:32:44.355428 24455 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e: No blocks or log segments found. Creating new log. I0107 12:32:44.355579 24455 log.cc:815] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e: Log is configured to *not* fsync() on all Append() calls I0107 12:32:44.356508 24455 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e: No bootstrap required, opened a new log I0107 12:32:44.357597 24455 raft_consensus.cc:341] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } I0107 12:32:44.357617 24455 raft_consensus.cc:367] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.357625 24455 raft_consensus.cc:712] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12eb4d7452f4af5abdaa23e9a3d5e7e, State: Initialized, Role: FOLLOWER I0107 12:32:44.357699 24455 consensus_queue.cc:229] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } I0107 12:32:44.357719 24455 raft_consensus.cc:381] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0107 12:32:44.357728 24455 raft_consensus.cc:477] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0107 12:32:44.357738 24455 raft_consensus.cc:2949] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.358057 24455 raft_consensus.cc:499] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } I0107 12:32:44.358093 24455 leader_election.cc:307] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c12eb4d7452f4af5abdaa23e9a3d5e7e; no voters: I0107 12:32:44.358180 24455 leader_election.cc:293] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [CANDIDATE]: Term 1 election: Requested vote from peers I0107 12:32:44.358194 24457 raft_consensus.cc:2696] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 1 FOLLOWER]: Leader election won for term 1 I0107 12:32:44.358326 24457 raft_consensus.cc:672] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [term 1 LEADER]: Becoming Leader. State: Replica: c12eb4d7452f4af5abdaa23e9a3d5e7e, State: Running, Role: LEADER I0107 12:32:44.358366 24457 consensus_queue.cc:206] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } I0107 12:32:44.358530 24459 sys_catalog.cc:340] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c12eb4d7452f4af5abdaa23e9a3d5e7e. Latest consensus state: current_term: 1 leader_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } } I0107 12:32:44.358544 24459 sys_catalog.cc:343] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: This master's current role is: LEADER I0107 12:32:44.358549 24455 sys_catalog.cc:340] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } } I0107 12:32:44.358561 24455 sys_catalog.cc:343] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: This master's current role is: LEADER I0107 12:32:44.358654 24458 sys_catalog.cc:340] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c12eb4d7452f4af5abdaa23e9a3d5e7e" member_type: VOTER } } I0107 12:32:44.358671 24458 sys_catalog.cc:343] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: This master's current role is: LEADER I0107 12:32:44.358839 24455 sys_catalog.cc:440] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e [sys.catalog]: configured and running, proceeding with master startup. I0107 12:32:44.358857 24461 catalog_manager.cc:1089] Loading table and tablet metadata into memory... I0107 12:32:44.359225 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master as pid 24394 I0107 12:32:44.359398 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.208.1:0 --local_ip_for_outbound_sockets=127.23.208.1 --webserver_interface=127.23.208.1 --webserver_port=0 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0107 12:32:44.359197 24461 catalog_manager.cc:1100] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:32:44.372776 24465 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:32:44.373224 24465 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:32:44.374313 24465 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:32:44.374375 24465 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.1 I0107 12:32:44.375737 24465 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.23.208.1 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24465 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:32:44.378772 24465 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:32:44.379956 24465 hybrid_clock.cc:186] HybridClock initialized: now 1578429164379944 us; error 17 us; skew 500 ppm I0107 12:32:44.380079 24465 fs_manager.cc:263] Metadata directory not provided I0107 12:32:44.380129 24465 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:32:44.380208 24465 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0107 12:32:44.380244 24465 server_base.cc:453] Attempting to create new FS layout instead I0107 12:32:44.381144 24465 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/instance: uuid: "914675d8717742d1ae5929637e95acb2" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.381319 24465 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/instance: uuid: "914675d8717742d1ae5929637e95acb2" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.382642 24465 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/data/block_manager_instance: No such file or directory (error 2) I0107 12:32:44.385938 24465 fs_manager.cc:518] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s I0107 12:32:44.386132 24465 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:32:44.386234 24465 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:32:44.387639 24465 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0107 12:32:44.387693 24465 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal uuid: "914675d8717742d1ae5929637e95acb2" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.387750 24465 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:32:44.392030 24461 catalog_manager.cc:984] Generated new certificate authority record I0107 12:32:44.392964 24461 catalog_manager.cc:1109] Loading token signing keys... I0107 12:32:44.399791 24461 catalog_manager.cc:4455] T 00000000000000000000000000000000 P c12eb4d7452f4af5abdaa23e9a3d5e7e: Generated new TSK 0 I0107 12:32:44.399870 24461 catalog_manager.cc:1119] Initializing in-progress tserver states... I0107 12:32:44.420352 24465 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:32:44.420572 24465 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0107 12:32:44.420588 24465 ts_tablet_manager.cc:415] Registered 0 tablets I0107 12:32:44.427021 24465 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.1:35732 I0107 12:32:44.427281 24465 webserver.cc:382] Webserver started at http://127.23.208.1:39731/ using document root and password file I0107 12:32:44.427417 24465 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb I0107 12:32:44.430733 24587 heartbeater.cc:322] Connected to a master server at 127.23.208.62:54889 I0107 12:32:44.430757 24587 heartbeater.cc:406] Registering TS with master... I0107 12:32:44.430804 24587 heartbeater.cc:455] Master 127.23.208.62:54889 requested a full tablet report, sending... I0107 12:32:44.431007 24422 ts_manager.cc:192] Registered new tserver with Master: 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732) I0107 12:32:44.431350 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24465 I0107 12:32:44.431574 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.208.2:0 --local_ip_for_outbound_sockets=127.23.208.2 --webserver_interface=127.23.208.2 --webserver_port=0 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0107 12:32:44.431807 24422 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.208.1:51766 WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:32:44.442068 24591 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:32:44.442175 24591 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:32:44.442975 24591 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:32:44.443006 24591 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.2 I0107 12:32:44.444027 24591 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/info.pb --webserver_interface=127.23.208.2 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24591 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:32:44.446072 24591 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:32:44.447211 24591 hybrid_clock.cc:186] HybridClock initialized: now 1578429164447202 us; error 17 us; skew 500 ppm I0107 12:32:44.447293 24591 fs_manager.cc:263] Metadata directory not provided I0107 12:32:44.447312 24591 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:32:44.447367 24591 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0107 12:32:44.447379 24591 server_base.cc:453] Attempting to create new FS layout instead I0107 12:32:44.448017 24591 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/instance: uuid: "c0dcf07e91a6459fb83f90f61d363e44" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.448141 24591 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/instance: uuid: "c0dcf07e91a6459fb83f90f61d363e44" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.448976 24591 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/data/block_manager_instance: No such file or directory (error 2) I0107 12:32:44.451221 24591 fs_manager.cc:518] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s I0107 12:32:44.451354 24591 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:32:44.451419 24591 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:32:44.452307 24591 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.452330 24591 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal uuid: "c0dcf07e91a6459fb83f90f61d363e44" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.452359 24591 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:32:44.473557 24591 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:32:44.473744 24591 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0107 12:32:44.473758 24591 ts_tablet_manager.cc:415] Registered 0 tablets I0107 12:32:44.479542 24591 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.2:57827 I0107 12:32:44.479754 24591 webserver.cc:382] Webserver started at http://127.23.208.2:48525/ using document root and password file I0107 12:32:44.479884 24591 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/info.pb I0107 12:32:44.482419 24712 heartbeater.cc:322] Connected to a master server at 127.23.208.62:54889 I0107 12:32:44.482439 24712 heartbeater.cc:406] Registering TS with master... I0107 12:32:44.482506 24712 heartbeater.cc:455] Master 127.23.208.62:54889 requested a full tablet report, sending... I0107 12:32:44.482650 24422 ts_manager.cc:192] Registered new tserver with Master: c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827) I0107 12:32:44.483124 24422 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.208.2:47609 I0107 12:32:44.483361 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24591 I0107 12:32:44.483569 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.208.3:0 --local_ip_for_outbound_sockets=127.23.208.3 --webserver_interface=127.23.208.3 --webserver_port=0 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:32:44.493711 24715 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:32:44.493814 24715 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:32:44.494596 24715 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:32:44.494627 24715 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.3 I0107 12:32:44.495645 24715 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/info.pb --webserver_interface=127.23.208.3 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24715 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:32:44.497712 24715 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:32:44.498888 24715 hybrid_clock.cc:186] HybridClock initialized: now 1578429164498879 us; error 16 us; skew 500 ppm I0107 12:32:44.498972 24715 fs_manager.cc:263] Metadata directory not provided I0107 12:32:44.498993 24715 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:32:44.499042 24715 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0107 12:32:44.499053 24715 server_base.cc:453] Attempting to create new FS layout instead I0107 12:32:44.499698 24715 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/instance: uuid: "76f8bc397d684ec2b99494eb2fee53c4" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.499824 24715 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/instance: uuid: "76f8bc397d684ec2b99494eb2fee53c4" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.500665 24715 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/data/block_manager_instance: No such file or directory (error 2) I0107 12:32:44.502912 24715 fs_manager.cc:518] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s I0107 12:32:44.503043 24715 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:32:44.503110 24715 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:32:44.504024 24715 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.504047 24715 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal uuid: "76f8bc397d684ec2b99494eb2fee53c4" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:32:44.504076 24715 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:32:44.517071 24715 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:32:44.517275 24715 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0107 12:32:44.517289 24715 ts_tablet_manager.cc:415] Registered 0 tablets I0107 12:32:44.522106 24715 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.3:49994 I0107 12:32:44.522318 24715 webserver.cc:382] Webserver started at http://127.23.208.3:46425/ using document root and password file I0107 12:32:44.522475 24715 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/info.pb I0107 12:32:44.524976 24836 heartbeater.cc:322] Connected to a master server at 127.23.208.62:54889 I0107 12:32:44.524992 24836 heartbeater.cc:406] Registering TS with master... I0107 12:32:44.525023 24836 heartbeater.cc:455] Master 127.23.208.62:54889 requested a full tablet report, sending... I0107 12:32:44.525163 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24715 I0107 12:32:44.525401 24422 ts_manager.cc:192] Registered new tserver with Master: 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:32:44.525960 24422 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.208.3:45015 I0107 12:32:44.528498 24384 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0107 12:32:44.533265 24422 catalog_manager.cc:1454] Servicing CreateTable request from {username='jenkins'} at 127.23.208.62:33005: name: "TestTable" schema { columns { name: "rand_key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "link_to" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "insert_ts" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "updated" type: BOOL is_key: false is_nullable: false read_default_value: "\000" encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU" } partition_schema { range_schema { columns { name: "rand_key" } } } W0107 12:32:44.533345 24422 catalog_manager.cc:1619] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0107 12:32:44.536792 24771 tablet_service.cc:1034] Processing CreateTablet for tablet 77e67457a0334f92a22f52fe739203dd (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0107 12:32:44.536813 24772 tablet_service.cc:1034] Processing CreateTablet for tablet 30019a15e66f4115b11a50dbcaf93a87 (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0107 12:32:44.536793 24773 tablet_service.cc:1034] Processing CreateTablet for tablet 16e58229a0c648eda40d4376454f2981 (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0107 12:32:44.536881 24771 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 77e67457a0334f92a22f52fe739203dd. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.537021 24772 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 30019a15e66f4115b11a50dbcaf93a87. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.537118 24773 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 16e58229a0c648eda40d4376454f2981. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.537356 24647 tablet_service.cc:1034] Processing CreateTablet for tablet 77e67457a0334f92a22f52fe739203dd (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0107 12:32:44.537447 24647 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 77e67457a0334f92a22f52fe739203dd. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.538725 24522 tablet_service.cc:1034] Processing CreateTablet for tablet 77e67457a0334f92a22f52fe739203dd (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0107 12:32:44.538789 24522 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 77e67457a0334f92a22f52fe739203dd. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.538822 24524 tablet_service.cc:1034] Processing CreateTablet for tablet 16e58229a0c648eda40d4376454f2981 (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0107 12:32:44.538884 24524 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 16e58229a0c648eda40d4376454f2981. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.538986 24523 tablet_service.cc:1034] Processing CreateTablet for tablet 30019a15e66f4115b11a50dbcaf93a87 (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0107 12:32:44.539001 24648 tablet_service.cc:1034] Processing CreateTablet for tablet 30019a15e66f4115b11a50dbcaf93a87 (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0107 12:32:44.539024 24523 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 30019a15e66f4115b11a50dbcaf93a87. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.539054 24648 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 30019a15e66f4115b11a50dbcaf93a87. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.539429 24649 tablet_service.cc:1034] Processing CreateTablet for tablet 16e58229a0c648eda40d4376454f2981 (table=TestTable [id=66958ee3a1114edb8734c7a1312ec803]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0107 12:32:44.539475 24649 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 16e58229a0c648eda40d4376454f2981. 1 dirs total, 0 dirs full, 0 dirs failed I0107 12:32:44.540032 24852 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Bootstrap starting. I0107 12:32:44.540103 24851 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Bootstrap starting. I0107 12:32:44.540320 24852 tablet_bootstrap.cc:590] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: No blocks or log segments found. Creating new log. I0107 12:32:44.540529 24852 log.cc:815] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Log is configured to *not* fsync() on all Append() calls I0107 12:32:44.540904 24854 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:32:44.541188 24851 tablet_bootstrap.cc:590] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: No blocks or log segments found. Creating new log. I0107 12:32:44.541239 24854 tablet_bootstrap.cc:590] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: No blocks or log segments found. Creating new log. I0107 12:32:44.541373 24851 log.cc:815] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Log is configured to *not* fsync() on all Append() calls I0107 12:32:44.541463 24854 log.cc:815] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Log is configured to *not* fsync() on all Append() calls I0107 12:32:44.541586 24852 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: No bootstrap required, opened a new log I0107 12:32:44.541620 24852 ts_tablet_manager.cc:1105] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0107 12:32:44.541963 24851 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: No bootstrap required, opened a new log I0107 12:32:44.541993 24851 ts_tablet_manager.cc:1105] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0107 12:32:44.542155 24854 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: No bootstrap required, opened a new log I0107 12:32:44.542177 24854 ts_tablet_manager.cc:1105] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0107 12:32:44.542945 24852 raft_consensus.cc:341] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.542960 24852 raft_consensus.cc:367] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.542968 24852 raft_consensus.cc:712] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dcf07e91a6459fb83f90f61d363e44, State: Initialized, Role: FOLLOWER I0107 12:32:44.543021 24851 raft_consensus.cc:341] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.543032 24851 raft_consensus.cc:367] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.543037 24851 raft_consensus.cc:712] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f8bc397d684ec2b99494eb2fee53c4, State: Initialized, Role: FOLLOWER I0107 12:32:44.543035 24852 consensus_queue.cc:229] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.543088 24851 consensus_queue.cc:229] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.543126 24852 ts_tablet_manager.cc:1133] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I0107 12:32:44.543145 24852 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Bootstrap starting. I0107 12:32:44.543174 24851 ts_tablet_manager.cc:1133] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0107 12:32:44.543177 24836 heartbeater.cc:447] Master 127.23.208.62:54889 was elected leader, sending a full tablet report... I0107 12:32:44.543192 24851 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Bootstrap starting. I0107 12:32:44.543363 24851 tablet_bootstrap.cc:590] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: No blocks or log segments found. Creating new log. I0107 12:32:44.543431 24852 tablet_bootstrap.cc:590] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: No blocks or log segments found. Creating new log. I0107 12:32:44.543550 24854 raft_consensus.cc:341] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.543562 24854 raft_consensus.cc:367] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.543570 24854 raft_consensus.cc:712] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Initialized, Role: FOLLOWER I0107 12:32:44.543622 24854 consensus_queue.cc:229] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.543633 24712 heartbeater.cc:447] Master 127.23.208.62:54889 was elected leader, sending a full tablet report... I0107 12:32:44.543704 24854 ts_tablet_manager.cc:1133] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0107 12:32:44.543725 24854 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:32:44.544006 24854 tablet_bootstrap.cc:590] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: No blocks or log segments found. Creating new log. I0107 12:32:44.544025 24587 heartbeater.cc:447] Master 127.23.208.62:54889 was elected leader, sending a full tablet report... I0107 12:32:44.544250 24851 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: No bootstrap required, opened a new log I0107 12:32:44.544268 24851 ts_tablet_manager.cc:1105] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0107 12:32:44.544348 24851 raft_consensus.cc:341] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.544354 24851 raft_consensus.cc:367] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.544358 24851 raft_consensus.cc:712] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f8bc397d684ec2b99494eb2fee53c4, State: Initialized, Role: FOLLOWER I0107 12:32:44.544395 24851 consensus_queue.cc:229] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.544411 24852 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: No bootstrap required, opened a new log I0107 12:32:44.544425 24852 ts_tablet_manager.cc:1105] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0107 12:32:44.544452 24851 ts_tablet_manager.cc:1133] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.544471 24851 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Bootstrap starting. I0107 12:32:44.544513 24852 raft_consensus.cc:341] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.544523 24852 raft_consensus.cc:367] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.544528 24852 raft_consensus.cc:712] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dcf07e91a6459fb83f90f61d363e44, State: Initialized, Role: FOLLOWER I0107 12:32:44.544569 24852 consensus_queue.cc:229] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.544603 24852 ts_tablet_manager.cc:1133] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.544620 24852 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Bootstrap starting. I0107 12:32:44.544648 24851 tablet_bootstrap.cc:590] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: No blocks or log segments found. Creating new log. I0107 12:32:44.544898 24852 tablet_bootstrap.cc:590] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: No blocks or log segments found. Creating new log. I0107 12:32:44.545199 24854 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: No bootstrap required, opened a new log I0107 12:32:44.545219 24854 ts_tablet_manager.cc:1105] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0107 12:32:44.545310 24854 raft_consensus.cc:341] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.545320 24854 raft_consensus.cc:367] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.545331 24854 raft_consensus.cc:712] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Initialized, Role: FOLLOWER I0107 12:32:44.545356 24854 consensus_queue.cc:229] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.545434 24854 ts_tablet_manager.cc:1133] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.545456 24854 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:32:44.545622 24854 tablet_bootstrap.cc:590] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: No blocks or log segments found. Creating new log. I0107 12:32:44.545918 24851 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: No bootstrap required, opened a new log I0107 12:32:44.545941 24851 ts_tablet_manager.cc:1105] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0107 12:32:44.546025 24852 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: No bootstrap required, opened a new log I0107 12:32:44.546025 24851 raft_consensus.cc:341] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.546038 24851 raft_consensus.cc:367] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.546041 24851 raft_consensus.cc:712] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f8bc397d684ec2b99494eb2fee53c4, State: Initialized, Role: FOLLOWER I0107 12:32:44.546046 24852 ts_tablet_manager.cc:1105] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I0107 12:32:44.546080 24851 consensus_queue.cc:229] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.546125 24851 ts_tablet_manager.cc:1133] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.546170 24852 raft_consensus.cc:341] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.546190 24852 raft_consensus.cc:367] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.546202 24852 raft_consensus.cc:712] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dcf07e91a6459fb83f90f61d363e44, State: Initialized, Role: FOLLOWER I0107 12:32:44.546255 24852 consensus_queue.cc:229] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.546340 24852 ts_tablet_manager.cc:1133] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.546511 24854 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: No bootstrap required, opened a new log I0107 12:32:44.546532 24854 ts_tablet_manager.cc:1105] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0107 12:32:44.546612 24854 raft_consensus.cc:341] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.546622 24854 raft_consensus.cc:367] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0107 12:32:44.546627 24854 raft_consensus.cc:712] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Initialized, Role: FOLLOWER I0107 12:32:44.546651 24854 consensus_queue.cc:229] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.546696 24854 ts_tablet_manager.cc:1133] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:32:44.605056 24857 raft_consensus.cc:477] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0107 12:32:44.605126 24857 raft_consensus.cc:499] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.605242 24857 leader_election.cc:293] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:32:44.607426 24793 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "30019a15e66f4115b11a50dbcaf93a87" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f8bc397d684ec2b99494eb2fee53c4" is_pre_election: true I0107 12:32:44.607462 24793 raft_consensus.cc:2396] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0dcf07e91a6459fb83f90f61d363e44 in term 0. I0107 12:32:44.607497 24544 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "30019a15e66f4115b11a50dbcaf93a87" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" is_pre_election: true I0107 12:32:44.607525 24544 raft_consensus.cc:2396] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0dcf07e91a6459fb83f90f61d363e44 in term 0. I0107 12:32:44.607605 24600 leader_election.cc:307] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: I0107 12:32:44.607652 24857 raft_consensus.cc:2696] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0107 12:32:44.607664 24857 raft_consensus.cc:477] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0107 12:32:44.607669 24857 raft_consensus.cc:2949] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.608148 24857 raft_consensus.cc:499] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.608183 24857 leader_election.cc:293] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 1 election: Requested vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:32:44.608280 24793 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "30019a15e66f4115b11a50dbcaf93a87" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f8bc397d684ec2b99494eb2fee53c4" I0107 12:32:44.608296 24544 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "30019a15e66f4115b11a50dbcaf93a87" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" I0107 12:32:44.608304 24793 raft_consensus.cc:2949] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.608309 24544 raft_consensus.cc:2949] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.609323 24544 raft_consensus.cc:2396] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0dcf07e91a6459fb83f90f61d363e44 in term 1. I0107 12:32:44.609401 24793 raft_consensus.cc:2396] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0dcf07e91a6459fb83f90f61d363e44 in term 1. I0107 12:32:44.609410 24600 leader_election.cc:307] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: I0107 12:32:44.609462 24857 raft_consensus.cc:2696] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Leader election won for term 1 I0107 12:32:44.609587 24858 raft_consensus.cc:477] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0107 12:32:44.609577 24857 raft_consensus.cc:672] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 LEADER]: Becoming Leader. State: Replica: c0dcf07e91a6459fb83f90f61d363e44, State: Running, Role: LEADER I0107 12:32:44.609629 24858 raft_consensus.cc:499] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.609652 24857 consensus_queue.cc:206] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.609705 24858 leader_election.cc:293] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732) I0107 12:32:44.610110 24422 catalog_manager.cc:4143] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 reported cstate change: term changed from 0 to 1, leader changed from to c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2). New cstate: current_term: 1 leader_uuid: "c0dcf07e91a6459fb83f90f61d363e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } health_report { overall_health: UNKNOWN } } } I0107 12:32:44.611833 24669 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" is_pre_election: true I0107 12:32:44.611858 24669 raft_consensus.cc:2396] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f8bc397d684ec2b99494eb2fee53c4 in term 0. I0107 12:32:44.611943 24725 leader_election.cc:307] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76f8bc397d684ec2b99494eb2fee53c4, c0dcf07e91a6459fb83f90f61d363e44; no voters: I0107 12:32:44.611989 24858 raft_consensus.cc:2696] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0107 12:32:44.612006 24858 raft_consensus.cc:477] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0107 12:32:44.612013 24858 raft_consensus.cc:2949] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.612028 24544 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" is_pre_election: true I0107 12:32:44.612051 24544 raft_consensus.cc:2396] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f8bc397d684ec2b99494eb2fee53c4 in term 0. I0107 12:32:44.612673 24858 raft_consensus.cc:499] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.612713 24858 leader_election.cc:293] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 1 election: Requested vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732) I0107 12:32:44.612816 24544 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" I0107 12:32:44.612821 24669 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" I0107 12:32:44.612831 24544 raft_consensus.cc:2949] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.612834 24669 raft_consensus.cc:2949] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.613291 24544 raft_consensus.cc:2396] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f8bc397d684ec2b99494eb2fee53c4 in term 1. I0107 12:32:44.613373 24724 leader_election.cc:307] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2; no voters: I0107 12:32:44.613409 24858 raft_consensus.cc:2696] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Leader election won for term 1 I0107 12:32:44.613489 24669 raft_consensus.cc:2396] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f8bc397d684ec2b99494eb2fee53c4 in term 1. I0107 12:32:44.613508 24858 raft_consensus.cc:672] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 LEADER]: Becoming Leader. State: Replica: 76f8bc397d684ec2b99494eb2fee53c4, State: Running, Role: LEADER I0107 12:32:44.613554 24858 consensus_queue.cc:206] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.613831 24422 catalog_manager.cc:4143] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 reported cstate change: term changed from 0 to 1, leader changed from to 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3). New cstate: current_term: 1 leader_uuid: "76f8bc397d684ec2b99494eb2fee53c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } health_report { overall_health: HEALTHY } } } W0107 12:32:44.677778 24588 tablet.cc:1792] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Can't schedule compaction. Clean time has not been advanced past its initial value. W0107 12:32:44.730223 24713 tablet.cc:1792] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Can't schedule compaction. Clean time has not been advanced past its initial value. W0107 12:32:44.772817 24837 tablet.cc:1792] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Can't schedule compaction. Clean time has not been advanced past its initial value. I0107 12:32:44.894281 24861 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0107 12:32:44.894347 24861 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.894481 24861 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:32:44.896386 24669 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" is_pre_election: true I0107 12:32:44.896416 24669 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 0. I0107 12:32:44.896538 24476 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: I0107 12:32:44.896585 24861 raft_consensus.cc:2696] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0107 12:32:44.896595 24861 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0107 12:32:44.896602 24861 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.896601 24793 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f8bc397d684ec2b99494eb2fee53c4" is_pre_election: true I0107 12:32:44.896620 24793 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 0. I0107 12:32:44.897421 24861 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.897464 24861 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 1 election: Requested vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:32:44.897562 24793 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f8bc397d684ec2b99494eb2fee53c4" I0107 12:32:44.897578 24793 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.897585 24669 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" I0107 12:32:44.897598 24669 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 0 FOLLOWER]: Advancing to term 1 I0107 12:32:44.898187 24793 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 1. I0107 12:32:44.898269 24475 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2; no voters: I0107 12:32:44.898310 24861 raft_consensus.cc:2696] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Leader election won for term 1 I0107 12:32:44.898344 24669 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 1. I0107 12:32:44.898403 24861 raft_consensus.cc:672] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 LEADER]: Becoming Leader. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Running, Role: LEADER I0107 12:32:44.898460 24861 consensus_queue.cc:206] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:32:44.898838 24422 catalog_manager.cc:4143] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 reported cstate change: term changed from 0 to 1, leader changed from to 914675d8717742d1ae5929637e95acb2 (127.23.208.1). New cstate: current_term: 1 leader_uuid: "914675d8717742d1ae5929637e95acb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } health_report { overall_health: UNKNOWN } } } I0107 12:32:44.971418 24384 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0107 12:32:44.971712 24384 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 914675d8717742d1ae5929637e95acb2 to finish bootstrapping I0107 12:32:44.973525 24384 ts_itest-base.cc:233] Waiting for 3 tablets on tserver c0dcf07e91a6459fb83f90f61d363e44 to finish bootstrapping I0107 12:32:44.974985 24384 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 76f8bc397d684ec2b99494eb2fee53c4 to finish bootstrapping I0107 12:32:44.977155 24384 linked_list-test-util.h:512] Running for: 900.000s I0107 12:32:44.977172 24384 linked_list-test-util.h:513] Sampling every 300000000 us I0107 12:32:44.977185 24384 linked_list-test-util.h:519] Written 0 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 0 Mean: 0 Percentiles: 0% (min) = 0 25% = 0 50% (med) = 0 75% = 0 95% = 0 99% = 0 99.9% = 0 99.99% = 0 100% (max) = 0 I0107 12:32:44.977927 24875 linked_list-test-util.h:379] Curl thread will poll the following URLs every 1000 ms: I0107 12:32:44.977980 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/maintenance-manager I0107 12:32:44.978116 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/metrics I0107 12:32:44.978389 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/maintenance-manager I0107 12:32:44.978510 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/tablets I0107 12:32:44.978555 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/tablets I0107 12:32:44.978595 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/threadz I0107 12:32:44.978634 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/metrics I0107 12:32:44.978672 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/mem-trackers I0107 12:32:44.978708 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/threadz?group=all I0107 12:32:44.978749 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/scans I0107 12:32:44.978786 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/threadz I0107 12:32:44.978826 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/threadz I0107 12:32:44.978863 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/masters I0107 12:32:44.978902 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/metrics I0107 12:32:44.978940 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/stacks I0107 12:32:44.978976 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/maintenance-manager I0107 12:32:44.979013 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/dump-entities I0107 12:32:44.979050 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/mem-trackers I0107 12:32:44.979087 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/scans I0107 12:32:44.979125 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/tables I0107 12:32:44.979162 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/transactions?tablet_id=77e67457a0334f92a22f52fe739203dd I0107 12:32:44.979199 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/threadz?group=all I0107 12:32:44.979235 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/transactions?tablet_id=77e67457a0334f92a22f52fe739203dd I0107 12:32:44.979274 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/scans I0107 12:32:44.979312 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/stacks I0107 12:32:44.979319 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/transactions?tablet_id=77e67457a0334f92a22f52fe739203dd I0107 12:32:44.979323 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/threadz?group=all I0107 12:32:44.979326 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/tablets I0107 12:32:44.979329 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/tablet-servers I0107 12:32:44.979332 24875 linked_list-test-util.h:382] http://127.23.208.1:39731/threadz I0107 12:32:44.979336 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/stacks I0107 12:32:44.979339 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/stacks I0107 12:32:44.979342 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/threadz?group=all I0107 12:32:44.979347 24875 linked_list-test-util.h:382] http://127.23.208.3:46425/mem-trackers I0107 12:32:44.979349 24875 linked_list-test-util.h:382] http://127.23.208.2:48525/mem-trackers I0107 12:32:44.979352 24875 linked_list-test-util.h:382] http://127.23.208.62:40214/metrics I0107 12:32:44.982359 24544 raft_consensus.cc:1195] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Refusing update from remote peer 76f8bc397d684ec2b99494eb2fee53c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0107 12:32:44.982553 24870 consensus_queue.cc:982] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0107 12:32:44.982635 24669 raft_consensus.cc:1195] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Refusing update from remote peer 76f8bc397d684ec2b99494eb2fee53c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0107 12:32:44.982766 24870 consensus_queue.cc:982] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0107 12:32:44.984083 24793 raft_consensus.cc:1195] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Refusing update from remote peer 914675d8717742d1ae5929637e95acb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0107 12:32:44.984093 24792 raft_consensus.cc:1195] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Refusing update from remote peer c0dcf07e91a6459fb83f90f61d363e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0107 12:32:44.984122 24881 mvcc.cc:203] Tried to move safe_time back from 6465245859766624256 to 6465245858257256448. Current Snapshot: MvccSnapshot[committed={T|T < 6465245859766624256}] I0107 12:32:44.984222 24857 consensus_queue.cc:982] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0107 12:32:44.984272 24873 consensus_queue.cc:982] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0107 12:32:44.984346 24668 raft_consensus.cc:1195] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Refusing update from remote peer 914675d8717742d1ae5929637e95acb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0107 12:32:44.984480 24861 consensus_queue.cc:982] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0107 12:32:44.984894 24544 raft_consensus.cc:1195] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Refusing update from remote peer c0dcf07e91a6459fb83f90f61d363e44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0107 12:32:44.985013 24857 consensus_queue.cc:982] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0107 12:32:44.985102 24883 mvcc.cc:203] Tried to move safe_time back from 6465245859766624256 to 6465245858257256448. Current Snapshot: MvccSnapshot[committed={T|T < 6465245859766624256 or (T in {6465245859766624256})}] I0107 12:32:44.985894 24882 mvcc.cc:203] Tried to move safe_time back from 6465245859766624256 to 6465245858257256448. Current Snapshot: MvccSnapshot[committed={T|T < 6465245859766624256}] ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 10000 Mean: 1569.57 Percentiles: 0% (min) = 800 25% = 1243 50% (med) = 1376 75% = 1546 95% = 1921 99% = 6136 99.9% = 35296 99.99% = 54944 100% (max) = 55186 I0107 12:33:02.483340 24384 linked_list-test-util.h:519] Written 500000 rows in chain I0107 12:33:07.824182 24879 thread.cc:387] Could not get per-thread statistics: Not found: /proc/self/task/25038/stat: No such file or directory (error 2) ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 20000 Mean: 1608.62 Percentiles: 0% (min) = 800 I0107 12:33:20.759850 24384 linked_list-test-util.h:519] Written 1000000 rows in chain 25% = 1271 50% (med) = 1408 75% = 1580 95% = 1990 99% = 6136 99.9% = 39904 99.99% = 55168 100% (max) = 61541 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 30000 Mean: 1623.98 Percentiles: 0% (min) = 800 25% = 1291 50% (med) = 1430 75% = 1603 95% = 2013 99% = 6184 99.9% = 34496 99.99% = 54944 100% (max) = 61541 I0107 12:33:39.325709 24384 linked_list-test-util.h:519] Written 1500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 40000 Mean: 1636.92 Percentiles: 0% (min) = 778 25% = 1303 50% (med) = 1445 75% = 1621 95% = 2029 99% = 6144 99.9% = 33824 99.99% = 53664 100% (max) = 61541 I0107 12:33:58.181879 24384 linked_list-test-util.h:519] Written 2000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 50000 Mean: 1643.33 Percentiles: 0% (min) = 778 I0107 12:34:16.811717 24384 linked_list-test-util.h:519] Written 2500000 rows in chain 25% = 1311 50% (med) = 1453 75% = 1631 95% = 2048 99% = 5968 99.9% = 33280 99.99% = 53184 100% (max) = 61541 I0107 12:34:35.452376 24384 linked_list-test-util.h:519] Written 3000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 60000 Mean: 1650.69 Percentiles: 0% (min) = 778 25% = 1314 50% (med) = 1456 75% = 1634 95% = 2058 99% = 6128 99.9% = 33824 99.99% = 53664 100% (max) = 72471 I0107 12:34:44.456341 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360 I0107 12:34:44.473153 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.002s sys 0.001s Metrics: {"bytes_written":7590,"cfile_init":1,"dirs.queue_time_us":9495,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":454,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":138,"lbm_writes_lt_1ms":27,"rows_written":6,"thread_start_us":63,"threads_started":2,"wal-append.queue_time_us":10843} I0107 12:34:44.473326 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 586 bytes on disk I0107 12:34:44.473538 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4} I0107 12:34:44.554139 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033336 I0107 12:34:44.575292 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033342 I0107 12:34:44.626047 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033356 I0107 12:34:47.544467 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.918s user 0.838s sys 0.062s Metrics: {"bytes_written":9588491,"cfile_init":1,"compiler_manager_pool.queue_time_us":6483,"compiler_manager_pool.run_cpu_time_us":173676,"compiler_manager_pool.run_wall_time_us":779766,"delete_count":0,"dirs.queue_time_us":3540,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":30130,"drs_written":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4,"lbm_write_time_us":119779,"lbm_writes_1-10_ms":18,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":954,"reinsert_count":0,"rows_written":540695,"thread_start_us":67,"threads_started":2,"update_count":1013} I0107 12:34:47.544634 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 41949923 bytes of WAL I0107 12:34:47.544740 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:34:47.544785 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000001 (ops 1-12467) I0107 12:34:47.544863 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000002 (ops 12468-24915) I0107 12:34:47.544924 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000003 (ops 24916-37374) I0107 12:34:47.545004 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000004 (ops 37375-49724) I0107 12:34:47.545060 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000005 (ops 49725-62170) I0107 12:34:47.620714 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.076s user 0.000s sys 0.018s Metrics: {} I0107 12:34:47.621857 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034188 I0107 12:34:48.080793 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 3.505s user 0.837s sys 0.079s Metrics: {"bytes_written":9580343,"cfile_init":1,"compiler_manager_pool.queue_time_us":6105,"compiler_manager_pool.run_cpu_time_us":178279,"compiler_manager_pool.run_wall_time_us":1159122,"delete_count":0,"dirs.queue_time_us":15299,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":45011,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":89379,"lbm_writes_1-10_ms":20,"lbm_writes_lt_1ms":954,"reinsert_count":0,"rows_written":540417,"thread_start_us":69,"threads_started":2,"update_count":735} I0107 12:34:48.081507 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 41947582 bytes of WAL I0107 12:34:48.081588 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:34:48.081640 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000001 (ops 1-12410) I0107 12:34:48.081720 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000002 (ops 12411-24835) I0107 12:34:48.082478 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000003 (ops 24836-37262) I0107 12:34:48.082548 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000004 (ops 37263-49635) I0107 12:34:48.082619 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000005 (ops 49636-62045) I0107 12:34:48.158742 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.076s user 0.000s sys 0.017s Metrics: {} I0107 12:34:48.164680 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034339 I0107 12:34:48.189561 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 3.635s user 0.848s sys 0.061s Metrics: {"bytes_written":9574778,"cfile_init":1,"compiler_manager_pool.queue_time_us":6909,"compiler_manager_pool.run_cpu_time_us":170499,"compiler_manager_pool.run_wall_time_us":780914,"delete_count":0,"dirs.queue_time_us":3583,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":92196,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":108908,"lbm_writes_1-10_ms":24,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":948,"reinsert_count":0,"rows_written":540327,"thread_start_us":78,"threads_started":2,"update_count":645} I0107 12:34:48.189677 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 41966482 bytes of WAL I0107 12:34:48.191763 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:34:48.191817 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000001 (ops 1-12410) I0107 12:34:48.193217 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000002 (ops 12411-24829) I0107 12:34:48.193332 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000003 (ops 24830-37251) I0107 12:34:48.193452 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000004 (ops 37252-49635) I0107 12:34:48.195277 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000005 (ops 49636-62045) I0107 12:34:48.231251 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.039s user 0.000s sys 0.013s Metrics: {} I0107 12:34:48.232107 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 10736497 bytes on disk I0107 12:34:48.232273 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4} I0107 12:34:48.232388 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034357 I0107 12:34:51.234774 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 3.613s user 0.832s sys 0.077s Metrics: {"bytes_written":9803192,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":18975,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":40579,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":115849,"lbm_writes_1-10_ms":24,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":970,"reinsert_count":0,"rows_written":553250,"thread_start_us":4396,"threads_started":1,"update_count":763} I0107 12:34:51.234936 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41932750 bytes of WAL I0107 12:34:51.235704 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:34:51.235777 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000001 (ops 1-12452) I0107 12:34:51.235882 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000002 (ops 12453-24867) I0107 12:34:51.235944 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000003 (ops 24868-37299) I0107 12:34:51.236001 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000004 (ops 37300-49668) I0107 12:34:51.236054 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000005 (ops 49669-62095) I0107 12:34:51.260244 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.025s user 0.000s sys 0.012s Metrics: {} I0107 12:34:51.260671 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 11002852 bytes on disk I0107 12:34:51.261286 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0107 12:34:51.262135 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.035199 I0107 12:34:51.307817 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 3.143s user 0.810s sys 0.084s Metrics: {"bytes_written":9869062,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":533,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":25868,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":81306,"lbm_writes_1-10_ms":18,"lbm_writes_lt_1ms":980,"reinsert_count":0,"rows_written":555644,"thread_start_us":41,"threads_started":1,"update_count":3157} I0107 12:34:51.307991 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41995427 bytes of WAL I0107 12:34:51.308061 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:34:51.308105 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000001 (ops 1-12448) I0107 12:34:51.308187 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000002 (ops 12449-24841) I0107 12:34:51.308255 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000003 (ops 24842-37250) I0107 12:34:51.308322 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000004 (ops 37251-49667) I0107 12:34:51.308383 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000005 (ops 49668-62082) I0107 12:34:51.343690 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.036s user 0.004s sys 0.009s Metrics: {} I0107 12:34:51.343835 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.035222 I0107 12:34:51.694016 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 3.461s user 0.859s sys 0.071s Metrics: {"bytes_written":9880134,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1484,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":94966,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":115747,"lbm_writes_1-10_ms":25,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":973,"reinsert_count":0,"rows_written":555903,"thread_start_us":42,"threads_started":1,"update_count":3416} I0107 12:34:51.694190 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41928906 bytes of WAL I0107 12:34:51.694959 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:34:51.695010 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000001 (ops 1-12449) I0107 12:34:51.695297 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000002 (ops 12450-24882) I0107 12:34:51.695361 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000003 (ops 24883-37338) I0107 12:34:51.695821 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000004 (ops 37339-49728) I0107 12:34:51.696038 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000005 (ops 49729-62187) I0107 12:34:51.767879 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.073s user 0.000s sys 0.013s Metrics: {} I0107 12:34:51.768189 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.035339 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 70000 Mean: 1673.19 Percentiles: 0% (min) = 778 25% = 1319 50% (med) = 1463 75% = 1649 95% = 2124 99% = 6416 99.9% = 34496 99.99% = 54944 100% (max) = 72471 I0107 12:34:55.080562 24384 linked_list-test-util.h:519] Written 3500000 rows in chain I0107 12:35:07.987977 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 16.726s user 3.648s sys 0.295s Metrics: {"bytes_written":40285358,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":10193,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":386251,"drs_written":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8,"lbm_write_time_us":882090,"lbm_writes_1-10_ms":96,"lbm_writes_10-100_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3944,"reinsert_count":0,"rows_written":2269221,"thread_start_us":1290,"threads_started":2,"update_count":12114} I0107 12:35:07.988157 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 45081769 bytes on disk I0107 12:35:07.988310 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":20,"lbm_reads_lt_1ms":8} I0107 12:35:07.988358 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 10740758 bytes on disk I0107 12:35:07.988502 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0107 12:35:08.110054 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 16.342s user 3.637s sys 0.282s Metrics: {"bytes_written":40318978,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":30998,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":82679,"drs_written":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8,"lbm_write_time_us":464636,"lbm_writes_1-10_ms":110,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":3936,"reinsert_count":0,"rows_written":2276337,"spinlock_wait_cycles":30592,"thread_start_us":60,"threads_started":2,"update_count":3910} I0107 12:35:08.110260 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92623956 bytes of WAL I0107 12:35:08.110575 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:35:08.110631 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000001 (ops 1-5935) I0107 12:35:08.110718 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000002 (ops 5936-11909) I0107 12:35:08.111608 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000003 (ops 11910-17776) I0107 12:35:08.111686 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000004 (ops 17777-23713) I0107 12:35:08.111748 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000005 (ops 23714-29674) I0107 12:35:08.111821 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000006 (ops 29675-35559) I0107 12:35:08.111912 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000007 (ops 35560-41517) I0107 12:35:08.112388 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000008 (ops 41518-47448) I0107 12:35:08.112480 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000009 (ops 47449-53455) I0107 12:35:08.112583 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000010 (ops 53456-59328) I0107 12:35:08.112641 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000011 (ops 59329-65216) I0107 12:35:08.213065 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.102s user 0.003s sys 0.021s Metrics: {} I0107 12:35:08.213212 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 45296984 bytes on disk I0107 12:35:08.213577 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0107 12:35:08.213762 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 11034469 bytes on disk I0107 12:35:08.213893 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0107 12:35:08.238731 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92659022 bytes of WAL I0107 12:35:08.238782 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:35:08.238790 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000001 (ops 1-5929) I0107 12:35:08.238837 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000002 (ops 5930-11898) I0107 12:35:08.238865 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000003 (ops 11899-17763) I0107 12:35:08.238888 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000004 (ops 17764-23713) I0107 12:35:08.238914 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000005 (ops 23714-29674) I0107 12:35:08.238935 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000006 (ops 29675-35559) I0107 12:35:08.238965 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000007 (ops 35560-41516) I0107 12:35:08.239058 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000008 (ops 41517-47455) I0107 12:35:08.239085 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000009 (ops 47456-53462) I0107 12:35:08.239112 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000010 (ops 53463-59333) I0107 12:35:08.239161 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000011 (ops 59334-65215) I0107 12:35:08.283629 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.045s user 0.002s sys 0.025s Metrics: {} I0107 12:35:08.392961 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 17.049s user 3.741s sys 0.306s Metrics: {"bytes_written":40318068,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":8034,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":235118,"drs_written":2,"lbm_read_time_us":171367,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":7,"lbm_write_time_us":533769,"lbm_writes_1-10_ms":107,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":3931,"reinsert_count":0,"rows_written":2270353,"thread_start_us":59,"threads_started":2,"update_count":13246} I0107 12:35:08.393079 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92572691 bytes of WAL I0107 12:35:08.393190 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:35:08.393242 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000001 (ops 1-5945) I0107 12:35:08.393981 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000002 (ops 5946-11921) I0107 12:35:08.394057 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000003 (ops 11922-17792) I0107 12:35:08.394121 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000004 (ops 17793-23713) I0107 12:35:08.394181 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000005 (ops 23714-29676) I0107 12:35:08.394421 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000006 (ops 29677-35559) I0107 12:35:08.394569 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000007 (ops 35560-41515) I0107 12:35:08.394670 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000008 (ops 41516-47453) I0107 12:35:08.394739 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000009 (ops 47454-53466) I0107 12:35:08.395177 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000010 (ops 53467-59343) I0107 12:35:08.395664 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000011 (ops 59344-65220) I0107 12:35:08.451308 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.058s user 0.001s sys 0.021s Metrics: {} I0107 12:35:08.451442 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 45094433 bytes on disk I0107 12:35:08.451628 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} I0107 12:35:08.451712 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 11031134 bytes on disk I0107 12:35:08.451825 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0107 12:35:08.451905 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 10737730 bytes on disk I0107 12:35:08.451998 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 80000 Mean: 1701.11 Percentiles: 0% (min) = 778 25% = 1330 50% (med) = 1479 75% = 1674 95% = 2198 I0107 12:35:15.505225 24384 linked_list-test-util.h:519] Written 4000000 rows in chain 99% = 6652 99.9% = 35104 99.99% = 60096 100% (max) = 72471 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 90000 Mean: 1712.12 Percentiles: 0% (min) = 778 25% = 1337 50% (med) = 1487 75% = 1684 95% = 2208 99% = 6724 99.9% = 36192 99.99% = 60096 100% (max) = 72471 I0107 12:35:35.219805 24384 linked_list-test-util.h:519] Written 4500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 100000 Mean: 1720.1 Percentiles: 0% (min) = 778 25% = 1344 50% (med) = 1495 75% = 1691 95% = 2212 99% = 6724 99.9% = 37184 99.99% = 61120 100% (max) = 72471 I0107 12:35:55.038285 24384 linked_list-test-util.h:519] Written 5000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 110000 Mean: 1728.54 Percentiles: 0% (min) = 778 25% = 1351 50% (med) = 1505 75% = 1703 95% = 2216 99% = 6728 99.9% = 37344 99.99% = 61024 100% (max) = 72471 I0107 12:36:15.148396 24384 linked_list-test-util.h:519] Written 5500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 120000 I0107 12:36:35.237025 24384 linked_list-test-util.h:519] Written 6000000 rows in chain Mean: 1736.29 Percentiles: 0% (min) = 778 25% = 1359 50% (med) = 1514 75% = 1714 95% = 2220 99% = 6732 99.9% = 37280 99.99% = 60096 100% (max) = 72471 I0107 12:36:47.642865 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033361 I0107 12:36:48.285410 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033390 I0107 12:36:48.300432 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033364 I0107 12:36:50.091869 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.449s user 0.759s sys 0.042s Metrics: {"bytes_written":9020214,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4908,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":39865,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":81471,"lbm_writes_1-10_ms":7,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":913,"reinsert_count":0,"rows_written":510865,"thread_start_us":29,"threads_started":1,"update_count":3875} I0107 12:36:50.092065 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33596101 bytes of WAL I0107 12:36:50.092133 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:36:50.092177 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000006 (ops 62171-74588) I0107 12:36:50.092273 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000007 (ops 74589-87047) I0107 12:36:50.092344 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000008 (ops 87048-99447) I0107 12:36:50.092432 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000009 (ops 99448-111856) I0107 12:36:50.136204 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.044s user 0.004s sys 0.007s Metrics: {} I0107 12:36:50.137389 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.004412 I0107 12:36:51.473775 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 3.173s user 0.787s sys 0.056s Metrics: {"bytes_written":9049468,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4734,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":57160,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":141727,"lbm_writes_1-10_ms":27,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":897,"reinsert_count":0,"rows_written":513504,"thread_start_us":32,"threads_started":1,"update_count":2083} I0107 12:36:51.476080 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33544650 bytes of WAL I0107 12:36:51.476162 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:36:51.476217 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000006 (ops 62046-74468) I0107 12:36:51.476296 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000007 (ops 74469-86821) I0107 12:36:51.476387 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000008 (ops 86822-99147) I0107 12:36:51.476495 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000009 (ops 99148-111524) I0107 12:36:51.503011 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.027s user 0.000s sys 0.011s Metrics: {} I0107 12:36:51.503162 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.004391 I0107 12:36:51.762307 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 3.476s user 0.769s sys 0.068s Metrics: {"bytes_written":9047427,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":22854,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":16019,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":133383,"lbm_writes_1-10_ms":28,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":897,"reinsert_count":0,"rows_written":513366,"thread_start_us":31,"threads_started":1,"update_count":2035} I0107 12:36:51.762512 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33565897 bytes of WAL I0107 12:36:51.762584 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:36:51.762629 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000006 (ops 62046-74470) I0107 12:36:51.762702 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000007 (ops 74471-86831) I0107 12:36:51.762933 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000008 (ops 86832-99199) I0107 12:36:51.762997 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000009 (ops 99200-111582) I0107 12:36:51.791841 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.029s user 0.002s sys 0.008s Metrics: {} I0107 12:36:51.792008 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 10187956 bytes on disk I0107 12:36:51.792136 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0107 12:36:51.792229 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.004390 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 130000 Mean: 1748.91 Percentiles: 0% (min) = 778 25% = 1365 50% (med) = 1521 75% = 1725 95% = 2246 99% = 6792 99.9% = 37344 99.99% = 61120 100% (max) = 245022 I0107 12:36:55.830487 24384 linked_list-test-util.h:519] Written 6500000 rows in chain I0107 12:36:56.445124 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 6.307s user 1.439s sys 0.134s Metrics: {"bytes_written":18459737,"cfile_cache_hit":50,"cfile_cache_hit_bytes":5922666,"cfile_cache_miss":2440,"cfile_cache_miss_bytes":67939970,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":12422,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":36916,"drs_written":1,"lbm_read_time_us":124469,"lbm_reads_1-10_ms":28,"lbm_reads_lt_1ms":2452,"lbm_write_time_us":193556,"lbm_writes_1-10_ms":40,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":1829,"num_input_rowsets":2,"rows_written":1051560,"spinlock_wait_cycles":12102528,"thread_start_us":446,"threads_started":9} I0107 12:36:56.445231 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034781 I0107 12:36:56.816131 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 5.024s user 1.263s sys 0.137s Metrics: {"bytes_written":18501718,"cfile_cache_hit":48,"cfile_cache_hit_bytes":5599389,"cfile_cache_miss":2448,"cfile_cache_miss_bytes":68413061,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":284,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":11577,"drs_written":1,"lbm_read_time_us":81958,"lbm_reads_1-10_ms":19,"lbm_reads_lt_1ms":2465,"lbm_write_time_us":197094,"lbm_writes_1-10_ms":43,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":1830,"num_input_rowsets":2,"rows_written":1053783,"thread_start_us":443,"threads_started":8} I0107 12:36:56.816339 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034863 I0107 12:36:57.158708 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 5.655s user 1.373s sys 0.155s Metrics: {"bytes_written":18502143,"cfile_cache_hit":52,"cfile_cache_hit_bytes":6133401,"cfile_cache_miss":2444,"cfile_cache_miss_bytes":67882395,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":67168,"drs_written":1,"lbm_read_time_us":149403,"lbm_reads_1-10_ms":18,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2465,"lbm_write_time_us":260029,"lbm_writes_1-10_ms":52,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":1819,"num_input_rowsets":2,"rows_written":1053831,"spinlock_wait_cycles":798208,"thread_start_us":388,"threads_started":8} I0107 12:36:57.162551 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034853 I0107 12:36:57.784701 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.968s user 0.586s sys 0.033s Metrics: {"bytes_written":9329890,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":137667,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":119906,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":948,"reinsert_count":0,"rows_written":529716,"thread_start_us":33,"threads_started":1,"update_count":2297} I0107 12:36:57.784932 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41950971 bytes of WAL I0107 12:36:57.785014 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:36:57.785054 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000006 (ops 62083-74487) I0107 12:36:57.785123 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000007 (ops 74488-86867) I0107 12:36:57.785181 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000008 (ops 86868-99257) I0107 12:36:57.785233 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000009 (ops 99258-111655) I0107 12:36:57.785286 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000010 (ops 111656-124073) I0107 12:36:57.798837 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.014s user 0.000s sys 0.013s Metrics: {} I0107 12:36:57.799055 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 21418112 bytes on disk I0107 12:36:57.886641 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.087s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87408,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0107 12:36:57.886762 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 10538719 bytes on disk I0107 12:36:57.886889 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0107 12:36:57.886945 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.004063 I0107 12:36:58.300513 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 1.855s user 0.660s sys 0.038s Metrics: {"bytes_written":9373928,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":42,"dirs.run_wall_time_us":43,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":954402,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":947,"reinsert_count":0,"rows_written":532052,"thread_start_us":113,"threads_started":1,"update_count":2248} I0107 12:36:58.300635 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41949929 bytes of WAL I0107 12:36:58.300719 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:36:58.300736 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000006 (ops 62096-74564) I0107 12:36:58.300776 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000007 (ops 74565-86964) I0107 12:36:58.300822 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000008 (ops 86965-99385) I0107 12:36:58.300875 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000009 (ops 99386-111806) I0107 12:36:58.300909 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000010 (ops 111807-124183) I0107 12:36:58.338425 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.038s user 0.002s sys 0.008s Metrics: {} I0107 12:36:58.338624 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 21369361 bytes on disk I0107 12:36:58.387789 26202 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=386034, attempt_no=0}) took 1067ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":10,"cfile_cache_miss":10,"cfile_cache_miss_bytes":45242,"cfile_init":1,"lbm_read_time_us":1066650,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":8,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":1,"num_ops":9,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":73,"raft.queue_time_us":49,"raft.run_cpu_time_us":34,"raft.run_wall_time_us":34,"replication_time_us":462}]]} I0107 12:36:58.433960 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.095s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95189,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":2} I0107 12:36:58.434046 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 10586147 bytes on disk I0107 12:36:58.486321 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.052s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52097,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3} I0107 12:36:58.486449 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.004064 I0107 12:36:58.873664 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 1.709s user 0.665s sys 0.045s Metrics: {"bytes_written":9302011,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":29815,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804444,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":11,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":939,"reinsert_count":0,"rows_written":529491,"thread_start_us":116,"threads_started":1,"update_count":43} I0107 12:36:58.873795 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41967847 bytes of WAL I0107 12:36:58.873884 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:36:58.873899 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000006 (ops 62188-74645) I0107 12:36:58.873939 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000007 (ops 74646-87151) I0107 12:36:58.873970 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000008 (ops 87152-99611) I0107 12:36:58.873991 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000009 (ops 99612-111951) I0107 12:36:58.874027 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000010 (ops 111952-124405) I0107 12:36:58.886893 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.013s user 0.001s sys 0.012s Metrics: {} I0107 12:36:58.887085 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.004063 I0107 12:37:03.484421 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 5.597s user 1.370s sys 0.168s Metrics: {"bytes_written":19045434,"cfile_cache_hit":63,"cfile_cache_hit_bytes":8131690,"cfile_cache_miss":2507,"cfile_cache_miss_bytes":68171388,"cfile_init":8,"delete_count":0,"delta_iterators_relevant":8,"dirs.queue_time_us":2360,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":39106,"drs_written":1,"lbm_read_time_us":1263761,"lbm_reads_1-10_ms":32,"lbm_reads_10-100_ms":19,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":2485,"lbm_write_time_us":198933,"lbm_writes_1-10_ms":38,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":1892,"num_input_rowsets":2,"reinsert_count":0,"rows_written":1085360,"spinlock_wait_cycles":4608,"thread_start_us":359,"threads_started":9,"update_count":9} I0107 12:37:03.488061 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 22115205 bytes on disk I0107 12:37:03.690490 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.202s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":202241,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":1} I0107 12:37:03.990705 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 5.504s user 1.414s sys 0.138s Metrics: {"bytes_written":19044599,"cfile_cache_hit":48,"cfile_cache_hit_bytes":5719041,"cfile_cache_miss":2522,"cfile_cache_miss_bytes":70570445,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":5101,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":73822,"drs_written":1,"lbm_read_time_us":1016181,"lbm_reads_1-10_ms":22,"lbm_reads_10-100_ms":19,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":2515,"lbm_write_time_us":206185,"lbm_writes_1-10_ms":45,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":1883,"num_input_rowsets":2,"rows_written":1085302,"spinlock_wait_cycles":3584,"thread_start_us":1180,"threads_started":9} I0107 12:37:03.990850 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 22114410 bytes on disk I0107 12:37:03.990953 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0107 12:37:04.771361 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 5.883s user 1.385s sys 0.170s Metrics: {"bytes_written":19045978,"cfile_cache_hit":53,"cfile_cache_hit_bytes":6540756,"cfile_cache_miss":2517,"cfile_cache_miss_bytes":69761168,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":505,"dirs.run_cpu_time_us":1602,"dirs.run_wall_time_us":81974,"drs_written":1,"lbm_read_time_us":1212372,"lbm_reads_1-10_ms":23,"lbm_reads_10-100_ms":24,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":2509,"lbm_write_time_us":249835,"lbm_writes_1-10_ms":35,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":1893,"num_input_rowsets":2,"rows_written":1085394,"spinlock_wait_cycles":18892032,"thread_start_us":527,"threads_started":8} I0107 12:37:04.771544 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 22116239 bytes on disk I0107 12:37:04.835109 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.063s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63428,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0107 12:37:04.835203 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 21419114 bytes on disk I0107 12:37:04.882669 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.047s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47298,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0107 12:37:07.996652 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033336 I0107 12:37:08.135326 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033340 I0107 12:37:08.444121 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033348 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 140000 Mean: 1792.91 Percentiles: 0% (min) = 778 25% = 1371 50% (med) = 1531 75% = 1742 95% = 2306 99% = 7012 99.9% = 39264 99.99% = 67648 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:37:20.857795 24384 linked_list-test-util.h:519] Written 7000000 rows in chain W0107 12:37:22.145277 24482 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.226s user 0.000s sys 0.000s I0107 12:37:22.146205 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 14.011s user 3.553s sys 0.219s Metrics: {"bytes_written":38634128,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":18127,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":307266,"drs_written":2,"lbm_read_time_us":13,"lbm_reads_lt_1ms":8,"lbm_write_time_us":813761,"lbm_writes_1-10_ms":85,"lbm_writes_10-100_ms":5,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3815,"reinsert_count":0,"rows_written":2194648,"thread_start_us":87,"threads_started":2,"update_count":9812} I0107 12:37:22.146320 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=1.008576 I0107 12:37:23.812055 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 15.815s user 3.491s sys 0.285s Metrics: {"bytes_written":38704479,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":13360,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":318186,"drs_written":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8,"lbm_write_time_us":887316,"lbm_writes_1-10_ms":99,"lbm_writes_10-100_ms":2,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3818,"reinsert_count":0,"rows_written":2199831,"thread_start_us":130,"threads_started":2,"update_count":7879} I0107 12:37:23.812273 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92619671 bytes of WAL I0107 12:37:23.812348 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:37:23.812389 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000012 (ops 65216-71067) I0107 12:37:23.812496 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000013 (ops 71068-76988) I0107 12:37:23.812564 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000014 (ops 76989-82889) I0107 12:37:23.812628 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000015 (ops 82890-88859) I0107 12:37:23.812687 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000016 (ops 88860-94722) I0107 12:37:23.812745 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000017 (ops 94723-100712) I0107 12:37:23.812803 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000018 (ops 100713-106579) I0107 12:37:23.812894 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000019 (ops 106580-112454) I0107 12:37:23.812953 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000020 (ops 112455-118417) I0107 12:37:23.813005 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000021 (ops 118418-124356) I0107 12:37:23.813060 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000022 (ops 124357-130257) I0107 12:37:23.947700 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.135s user 0.000s sys 0.030s Metrics: {} I0107 12:37:23.947837 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=1.008595 I0107 12:37:24.521988 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 16.078s user 3.593s sys 0.313s Metrics: {"bytes_written":38882946,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":16048,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":150701,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":475886,"lbm_writes_1-10_ms":124,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3804,"reinsert_count":0,"rows_written":2205774,"thread_start_us":68,"threads_started":2,"update_count":14954} I0107 12:37:24.522178 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92567378 bytes of WAL I0107 12:37:24.522253 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:37:24.522296 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000012 (ops 65221-71111) I0107 12:37:24.522444 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000013 (ops 71112-77043) I0107 12:37:24.522514 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000014 (ops 77044-82887) I0107 12:37:24.522716 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000015 (ops 82888-88871) I0107 12:37:24.522791 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000016 (ops 88872-94723) I0107 12:37:24.522867 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000017 (ops 94724-100719) I0107 12:37:24.522935 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000018 (ops 100720-106591) I0107 12:37:24.523138 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000019 (ops 106592-112460) I0107 12:37:24.523202 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000020 (ops 112461-118502) I0107 12:37:24.523407 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000021 (ops 118503-124410) I0107 12:37:24.523494 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000022 (ops 124411-130318) I0107 12:37:24.606127 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.084s user 0.000s sys 0.029s Metrics: {} I0107 12:37:24.606357 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=1.008522 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 150000 Mean: 1815.8 Percentiles: 0% (min) = 778 25% = 1377 50% (med) = 1541 75% = 1759 95% = 2370 99% = 7236 99.9% = 40064 99.99% = 72448 100% (max) = 1068091 I0107 12:37:43.616959 24384 linked_list-test-util.h:519] Written 7500000 rows in chain *NOTE: some values were greater than highest trackable value I0107 12:37:44.978018 24384 linked_list-test-util.h:528] Sample at HT timestamp: 6465247088543801344 Inserted count: 7530250 I0107 12:37:47.833643 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 25.686s user 6.169s sys 0.563s Metrics: {"bytes_written":78416322,"cfile_cache_hit":210,"cfile_cache_hit_bytes":28863484,"cfile_cache_miss":10334,"cfile_cache_miss_bytes":285675656,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":20252,"dirs.run_cpu_time_us":361,"dirs.run_wall_time_us":499225,"drs_written":3,"lbm_read_time_us":2638059,"lbm_reads_1-10_ms":154,"lbm_reads_10-100_ms":44,"lbm_reads_gt_100_ms":5,"lbm_reads_lt_1ms":10215,"lbm_write_time_us":1092515,"lbm_writes_1-10_ms":147,"lbm_writes_10-100_ms":6,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":7776,"num_input_rowsets":4,"rows_written":4470985,"spinlock_wait_cycles":36656000,"thread_start_us":223,"threads_started":3} I0107 12:37:47.833869 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92596825 bytes of WAL I0107 12:37:47.834062 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:37:47.834112 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000012 (ops 65217-71082) I0107 12:37:47.834249 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000013 (ops 71083-77006) I0107 12:37:47.834345 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000014 (ops 77007-82888) I0107 12:37:47.834410 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000015 (ops 82889-88860) I0107 12:37:47.834551 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000016 (ops 88861-94723) I0107 12:37:47.834620 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000017 (ops 94724-100715) I0107 12:37:47.834692 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000018 (ops 100716-106592) I0107 12:37:47.834764 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000019 (ops 106593-112457) I0107 12:37:47.834825 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000020 (ops 112458-118443) I0107 12:37:47.834905 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000021 (ops 118444-124356) I0107 12:37:47.834962 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000022 (ops 124357-130266) I0107 12:37:47.906669 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.073s user 0.000s sys 0.026s Metrics: {} I0107 12:37:47.906841 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 91286985 bytes on disk I0107 12:37:48.026326 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.119s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":119248,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":6} I0107 12:37:49.278425 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 25.322s user 5.928s sys 0.621s Metrics: {"bytes_written":78383688,"cfile_cache_hit":223,"cfile_cache_hit_bytes":31100984,"cfile_cache_miss":10335,"cfile_cache_miss_bytes":283337536,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":10531,"dirs.run_cpu_time_us":308,"dirs.run_wall_time_us":111227,"drs_written":3,"lbm_read_time_us":2816772,"lbm_reads_1-10_ms":146,"lbm_reads_10-100_ms":49,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":10218,"lbm_write_time_us":860856,"lbm_writes_1-10_ms":196,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":7726,"num_input_rowsets":4,"rows_written":4469052,"spinlock_wait_cycles":147773312,"thread_start_us":528,"threads_started":11} I0107 12:37:49.278617 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 91245730 bytes on disk I0107 12:37:49.471809 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.193s user 0.001s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":192924,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":6} I0107 12:37:49.587852 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 24.981s user 5.967s sys 0.580s Metrics: {"bytes_written":78504054,"cfile_cache_hit":229,"cfile_cache_hit_bytes":31609947,"cfile_cache_miss":10343,"cfile_cache_miss_bytes":283358549,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":24166,"dirs.run_cpu_time_us":431,"dirs.run_wall_time_us":120379,"drs_written":3,"lbm_read_time_us":3310385,"lbm_reads_1-10_ms":127,"lbm_reads_10-100_ms":49,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":10242,"lbm_write_time_us":945006,"lbm_writes_1-10_ms":182,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":7747,"num_input_rowsets":4,"rows_written":4476127,"spinlock_wait_cycles":122148864,"thread_start_us":583,"threads_started":11} I0107 12:37:49.587983 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 91393105 bytes on disk I0107 12:37:49.836410 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.248s user 0.000s sys 0.001s Metrics: {"cfile_init":3,"lbm_read_time_us":248210,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":6} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 160000 Mean: 1830.71 Percentiles: 0% (min) = 778 25% = 1381 50% (med) = 1545 75% = 1765 95% = 2380 99% = 7380 99.9% = 41504 99.99% = 99328 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:38:05.886168 24384 linked_list-test-util.h:519] Written 8000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 170000 Mean: 1832.78 Percentiles: 0% (min) = 778 25% = 1386 50% (med) = 1550 75% = 1769 95% = 2374 99% = 7376 99.9% = 41088 99.99% = 99328 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:38:26.516680 24384 linked_list-test-util.h:519] Written 8500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 180000 Mean: 1833.79 Percentiles: 0% (min) = 778 25% = 1390 50% (med) = 1555 75% = 1773 95% = 2370 99% = 7352 99.9% = 40320 99.99% = 97088 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:38:46.976130 24384 linked_list-test-util.h:519] Written 9000000 rows in chain I0107 12:38:50.278090 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033385 I0107 12:38:51.581873 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033363 I0107 12:38:51.829260 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 1.551s user 0.642s sys 0.052s Metrics: {"bytes_written":8217453,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11034,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":78694,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":43599,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":832,"reinsert_count":0,"rows_written":462580,"thread_start_us":58,"threads_started":1,"update_count":1605} I0107 12:38:51.829461 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 42070531 bytes of WAL I0107 12:38:51.829676 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:38:51.829727 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000010 (ops 111857-124240) I0107 12:38:51.829843 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000011 (ops 124241-136760) I0107 12:38:51.829916 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000012 (ops 136761-149159) I0107 12:38:51.829972 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000013 (ops 149160-161536) I0107 12:38:51.830459 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000014 (ops 161537-173962) I0107 12:38:51.874107 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.044s user 0.000s sys 0.014s Metrics: {} I0107 12:38:51.882371 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001324 I0107 12:38:51.892454 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033370 I0107 12:38:53.978026 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.394s user 0.709s sys 0.050s Metrics: {"bytes_written":8284183,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":291,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":69347,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":82341,"lbm_writes_1-10_ms":21,"lbm_writes_lt_1ms":822,"reinsert_count":0,"rows_written":465532,"thread_start_us":31,"threads_started":1,"update_count":3009} I0107 12:38:53.978257 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 41994042 bytes of WAL I0107 12:38:53.978603 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:38:53.978758 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000010 (ops 111525-123946) I0107 12:38:53.978832 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000011 (ops 123947-136382) I0107 12:38:53.979063 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000012 (ops 136383-148749) I0107 12:38:53.979125 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000013 (ops 148750-161136) I0107 12:38:53.979748 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000014 (ops 161137-173483) I0107 12:38:54.038623 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.060s user 0.000s sys 0.014s Metrics: {} I0107 12:38:54.049666 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001294 I0107 12:38:54.446091 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.554s user 0.720s sys 0.047s Metrics: {"bytes_written":8266595,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4095,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":75237,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":66812,"lbm_writes_1-10_ms":22,"lbm_writes_lt_1ms":822,"reinsert_count":0,"rows_written":466529,"thread_start_us":70,"threads_started":1,"update_count":15} I0107 12:38:54.446215 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 42076612 bytes of WAL I0107 12:38:54.446995 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:38:54.447047 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000010 (ops 111583-124005) I0107 12:38:54.447115 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000011 (ops 124006-136559) I0107 12:38:54.447177 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000012 (ops 136560-148906) I0107 12:38:54.447274 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000013 (ops 148907-161338) I0107 12:38:54.447343 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000014 (ops 161339-173705) I0107 12:38:54.476140 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.029s user 0.000s sys 0.013s Metrics: {} I0107 12:38:54.478016 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001287 I0107 12:38:59.812204 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 7.930s user 2.060s sys 0.188s Metrics: {"bytes_written":26838497,"cfile_cache_hit":71,"cfile_cache_hit_bytes":9327303,"cfile_cache_miss":3507,"cfile_cache_miss_bytes":97884989,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":6832,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":44065,"drs_written":1,"lbm_read_time_us":567354,"lbm_reads_1-10_ms":54,"lbm_reads_10-100_ms":14,"lbm_reads_lt_1ms":3475,"lbm_write_time_us":288607,"lbm_writes_1-10_ms":61,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":2622,"mutex_wait_us":6550,"num_input_rowsets":2,"rows_written":1514140,"spinlock_wait_cycles":25344,"thread_start_us":480,"threads_started":9} I0107 12:38:59.812304 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 31356652 bytes on disk I0107 12:38:59.812413 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0107 12:38:59.812719 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.033753 I0107 12:39:02.310389 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 2.497s user 0.652s sys 0.048s Metrics: {"bytes_written":8232224,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6017,"dirs.run_cpu_time_us":59,"dirs.run_wall_time_us":89,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":610766,"lbm_writes_1-10_ms":6,"lbm_writes_10-100_ms":7,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":823,"reinsert_count":0,"rows_written":462470,"thread_start_us":42,"threads_started":1,"update_count":2973} I0107 12:39:02.310967 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 33693027 bytes of WAL I0107 12:39:02.311043 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 4 log segments from log reader I0107 12:39:02.311131 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000011 (ops 124184-136805) I0107 12:39:02.311236 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000012 (ops 136806-149291) I0107 12:39:02.311327 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000013 (ops 149292-161671) I0107 12:39:02.312304 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000014 (ops 161672-174077) I0107 12:39:02.375725 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.065s user 0.000s sys 0.013s Metrics: {} I0107 12:39:02.376047 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 9179325 bytes on disk I0107 12:39:02.382944 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":6151,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0107 12:39:02.383087 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.001330 I0107 12:39:02.953150 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 8.475s user 2.047s sys 0.207s Metrics: {"bytes_written":26948338,"cfile_cache_hit":73,"cfile_cache_hit_bytes":9376225,"cfile_cache_miss":3519,"cfile_cache_miss_bytes":98263889,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":4230,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":36776,"drs_written":1,"lbm_read_time_us":1044313,"lbm_reads_1-10_ms":60,"lbm_reads_10-100_ms":27,"lbm_reads_lt_1ms":3468,"lbm_write_time_us":251043,"lbm_writes_1-10_ms":45,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":2650,"num_input_rowsets":2,"rows_written":1520312,"spinlock_wait_cycles":50415232,"thread_start_us":289,"threads_started":9} I0107 12:39:02.953339 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 31485542 bytes on disk I0107 12:39:02.956485 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0107 12:39:02.956688 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034770 I0107 12:39:03.310052 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 9.260s user 2.018s sys 0.217s Metrics: {"bytes_written":26928822,"cfile_cache_hit":72,"cfile_cache_hit_bytes":9214957,"cfile_cache_miss":3518,"cfile_cache_miss_bytes":98376531,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":97003,"drs_written":1,"lbm_read_time_us":1304266,"lbm_reads_1-10_ms":42,"lbm_reads_10-100_ms":29,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3482,"lbm_write_time_us":279402,"lbm_writes_1-10_ms":62,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2630,"num_input_rowsets":2,"rows_written":1519363,"spinlock_wait_cycles":19840384,"thread_start_us":1422,"threads_started":9} I0107 12:39:03.310227 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034566 I0107 12:39:04.552110 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 1.592s user 0.583s sys 0.036s Metrics: {"bytes_written":8396330,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":131771,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":69967,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":843,"reinsert_count":0,"rows_written":471928,"thread_start_us":34,"threads_started":1,"update_count":3184} I0107 12:39:04.552949 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 33749445 bytes of WAL I0107 12:39:04.553524 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 4 log segments from log reader I0107 12:39:04.553580 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000011 (ops 124074-136748) I0107 12:39:04.553661 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000012 (ops 136749-149110) I0107 12:39:04.553864 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000013 (ops 149111-161533) I0107 12:39:04.553932 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000014 (ops 161534-173903) I0107 12:39:04.627189 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.074s user 0.000s sys 0.011s Metrics: {} I0107 12:39:04.627701 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.001234 I0107 12:39:05.915658 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 2.601s user 0.709s sys 0.053s Metrics: {"bytes_written":8393360,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":3713,"dirs.run_cpu_time_us":55,"dirs.run_wall_time_us":55,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":538706,"lbm_writes_1-10_ms":16,"lbm_writes_10-100_ms":6,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":832,"reinsert_count":0,"rows_written":473072,"thread_start_us":29,"threads_started":1,"update_count":1004} I0107 12:39:05.915879 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 33591451 bytes of WAL I0107 12:39:05.915957 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 4 log segments from log reader I0107 12:39:05.916000 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000011 (ops 124406-136864) I0107 12:39:05.916064 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000012 (ops 136865-149342) I0107 12:39:05.916121 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000013 (ops 149343-161741) I0107 12:39:05.916177 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000014 (ops 161742-174150) I0107 12:39:05.940604 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.025s user 0.001s sys 0.009s Metrics: {} I0107 12:39:05.941228 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.001224 I0107 12:39:10.223819 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 7.841s user 2.076s sys 0.217s Metrics: {"bytes_written":27428131,"cfile_cache_hit":74,"cfile_cache_hit_bytes":9755738,"cfile_cache_miss":3582,"cfile_cache_miss_bytes":99917546,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":105,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":120944,"drs_written":1,"lbm_read_time_us":1133798,"lbm_reads_1-10_ms":53,"lbm_reads_10-100_ms":20,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":3538,"lbm_write_time_us":248677,"lbm_writes_1-10_ms":49,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2694,"num_input_rowsets":2,"rows_written":1547772,"spinlock_wait_cycles":21641344,"thread_start_us":591,"threads_started":9} I0107 12:39:10.223979 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 32113240 bytes on disk I0107 12:39:10.224112 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 190000 Mean: 1854.3 Percentiles: 0% (min) = 778 25% = 1394 50% (med) = 1562 75% = 1787 95% = 2408 99% = 7480 99.9% = 41184 99.99% = 113728 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:39:10.697863 24384 linked_list-test-util.h:519] Written 9500000 rows in chain I0107 12:39:12.624402 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 7.997s user 2.067s sys 0.244s Metrics: {"bytes_written":27593289,"cfile_cache_hit":75,"cfile_cache_hit_bytes":9782373,"cfile_cache_miss":3605,"cfile_cache_miss_bytes":100563229,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":157,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":42966,"drs_written":1,"lbm_read_time_us":1026487,"lbm_reads_1-10_ms":54,"lbm_reads_10-100_ms":18,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":3571,"lbm_write_time_us":243695,"lbm_writes_1-10_ms":46,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":2714,"num_input_rowsets":2,"rows_written":1557288,"spinlock_wait_cycles":164096,"thread_start_us":1543,"threads_started":9} I0107 12:39:12.624519 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 32321722 bytes on disk I0107 12:39:12.624603 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0107 12:39:13.404562 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 7.463s user 2.014s sys 0.209s Metrics: {"bytes_written":27615705,"cfile_cache_hit":71,"cfile_cache_hit_bytes":9300483,"cfile_cache_miss":3611,"cfile_cache_miss_bytes":101117769,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":10392,"dirs.run_cpu_time_us":3424,"dirs.run_wall_time_us":32845,"drs_written":1,"lbm_read_time_us":1368216,"lbm_reads_1-10_ms":36,"lbm_reads_10-100_ms":27,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":3581,"lbm_write_time_us":208340,"lbm_writes_1-10_ms":41,"lbm_writes_lt_1ms":2722,"num_input_rowsets":2,"rows_written":1558466,"spinlock_wait_cycles":29828608,"thread_start_us":3794,"threads_started":9} I0107 12:39:13.404726 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 32349461 bytes on disk I0107 12:39:13.466449 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.062s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61555,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":2} I0107 12:39:13.466601 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 31465213 bytes on disk I0107 12:39:13.500607 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.034s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":33833,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0107 12:39:22.263073 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033366 I0107 12:39:24.002251 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033386 I0107 12:39:24.634202 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033365 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 200000 Mean: 1861.89 Percentiles: 0% (min) = 778 25% = 1398 50% (med) = 1567 75% = 1792 95% = 2420 99% = 7516 99.9% = 41888 99.99% = 113728 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:39:32.303555 24384 linked_list-test-util.h:519] Written 10000000 rows in chain I0107 12:39:34.288499 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 12.025s user 3.196s sys 0.266s Metrics: {"bytes_written":36277151,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":4575,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":56959,"drs_written":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8,"lbm_write_time_us":383653,"lbm_writes_1-10_ms":73,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3567,"reinsert_count":0,"rows_written":2042269,"thread_start_us":59,"threads_started":2,"update_count":12661} I0107 12:39:34.288636 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92232056 bytes of WAL I0107 12:39:34.288678 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:39:34.288691 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000023 (ops 130267-136234) I0107 12:39:34.288738 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000024 (ops 136235-142074) I0107 12:39:34.288769 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000025 (ops 142075-148046) I0107 12:39:34.288794 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000026 (ops 148047-153921) I0107 12:39:34.288828 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000027 (ops 153922-159805) I0107 12:39:34.288863 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000028 (ops 159806-165638) I0107 12:39:34.288934 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000029 (ops 165639-171533) I0107 12:39:34.289016 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000030 (ops 171534-177435) I0107 12:39:34.289050 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000031 (ops 177436-183333) I0107 12:39:34.289084 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000032 (ops 183334-189272) I0107 12:39:34.289156 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000033 (ops 189273-195151) I0107 12:39:34.322690 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.034s user 0.000s sys 0.024s Metrics: {} I0107 12:39:34.322837 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=1.007414 I0107 12:39:38.638579 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 14.636s user 3.312s sys 0.275s Metrics: {"bytes_written":36703215,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":818,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":457818,"drs_written":2,"lbm_read_time_us":409670,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":7,"lbm_write_time_us":476947,"lbm_writes_1-10_ms":112,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3577,"reinsert_count":0,"rows_written":2069621,"thread_start_us":66,"threads_started":2,"update_count":7650} I0107 12:39:38.638859 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92147967 bytes of WAL I0107 12:39:38.639009 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:39:38.639057 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000023 (ops 130258-136220) I0107 12:39:38.640354 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000024 (ops 136221-142038) I0107 12:39:38.640527 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000025 (ops 142039-147956) I0107 12:39:38.640599 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000026 (ops 147957-153795) I0107 12:39:38.640662 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000027 (ops 153796-159684) I0107 12:39:38.640720 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000028 (ops 159685-165575) I0107 12:39:38.640816 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000029 (ops 165576-171482) I0107 12:39:38.640913 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000030 (ops 171483-177387) I0107 12:39:38.640977 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000031 (ops 177388-183279) I0107 12:39:38.641268 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000032 (ops 183280-189215) I0107 12:39:38.642079 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000033 (ops 189216-195044) I0107 12:39:38.778426 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.139s user 0.001s sys 0.029s Metrics: {} I0107 12:39:38.778602 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=1.007149 I0107 12:39:39.515566 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 14.880s user 3.360s sys 0.299s Metrics: {"bytes_written":36838708,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":18844,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":117213,"drs_written":2,"lbm_read_time_us":13,"lbm_reads_lt_1ms":8,"lbm_write_time_us":525541,"lbm_writes_1-10_ms":104,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":3588,"reinsert_count":0,"rows_written":2071704,"thread_start_us":63,"threads_started":2,"update_count":16808} I0107 12:39:39.515765 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 92361335 bytes of WAL I0107 12:39:39.515942 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 11 log segments from log reader I0107 12:39:39.515995 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000023 (ops 130319-136294) I0107 12:39:39.516081 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000024 (ops 136295-142149) I0107 12:39:39.516150 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000025 (ops 142150-148127) I0107 12:39:39.516270 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000026 (ops 148128-154016) I0107 12:39:39.516340 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000027 (ops 154017-159905) I0107 12:39:39.516404 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000028 (ops 159906-165810) I0107 12:39:39.516908 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000029 (ops 165811-171724) I0107 12:39:39.517004 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000030 (ops 171725-177626) I0107 12:39:39.517755 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000031 (ops 177627-183524) I0107 12:39:39.517982 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000032 (ops 183525-189396) I0107 12:39:39.518137 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000033 (ops 189397-195409) I0107 12:39:39.641569 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.126s user 0.000s sys 0.027s Metrics: {} I0107 12:39:39.641710 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 41309318 bytes on disk I0107 12:39:39.643093 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0107 12:39:39.643182 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=1.007063 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 210000 Mean: 1877.67 Percentiles: 0% (min) = 778 25% = 1401 50% (med) = 1572 75% = 1803 95% = 2446 99% = 7644 99.9% = 42528 99.99% = 127616 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:39:55.402806 24384 linked_list-test-util.h:519] Written 10500000 rows in chain I0107 12:40:13.131745 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 38.809s user 8.700s sys 0.906s Metrics: {"bytes_written":115355765,"cfile_cache_hit":327,"cfile_cache_hit_bytes":45648695,"cfile_cache_miss":15039,"cfile_cache_miss_bytes":416355125,"cfile_init":23,"delta_iterators_relevant":14,"dirs.queue_time_us":7175,"dirs.run_cpu_time_us":553,"dirs.run_wall_time_us":238988,"drs_written":4,"lbm_read_time_us":5385808,"lbm_reads_1-10_ms":256,"lbm_reads_10-100_ms":84,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":14780,"lbm_write_time_us":1462806,"lbm_writes_1-10_ms":275,"lbm_writes_10-100_ms":11,"lbm_writes_lt_1ms":11258,"num_input_rowsets":5,"rows_written":6513254,"spinlock_wait_cycles":193665920,"thread_start_us":125,"threads_started":4} I0107 12:40:13.131925 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 135595937 bytes on disk I0107 12:40:13.385730 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.253s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":252782,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":9} I0107 12:40:15.847841 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 37.036s user 8.684s sys 0.870s Metrics: {"bytes_written":115800813,"cfile_cache_hit":343,"cfile_cache_hit_bytes":48069340,"cfile_cache_miss":15083,"cfile_cache_miss_bytes":415752496,"cfile_init":23,"delta_iterators_relevant":14,"dirs.queue_time_us":23558,"dirs.run_cpu_time_us":470,"dirs.run_wall_time_us":214185,"drs_written":4,"lbm_read_time_us":5745561,"lbm_reads_1-10_ms":199,"lbm_reads_10-100_ms":102,"lbm_reads_gt_100_ms":12,"lbm_reads_lt_1ms":14862,"lbm_write_time_us":1222498,"lbm_writes_1-10_ms":252,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":11327,"num_input_rowsets":5,"rows_written":6538673,"spinlock_wait_cycles":213508736,"thread_start_us":345,"threads_started":12} I0107 12:40:15.848026 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 136167836 bytes on disk I0107 12:40:15.848186 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":28,"lbm_reads_lt_1ms":16} I0107 12:40:16.264226 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 36.619s user 8.514s sys 0.912s Metrics: {"bytes_written":115960988,"cfile_cache_hit":347,"cfile_cache_hit_bytes":48537906,"cfile_cache_miss":15099,"cfile_cache_miss_bytes":415967760,"cfile_init":21,"delta_iterators_relevant":14,"dirs.queue_time_us":17531,"dirs.run_cpu_time_us":497,"dirs.run_wall_time_us":289590,"drs_written":4,"lbm_read_time_us":5942762,"lbm_reads_1-10_ms":201,"lbm_reads_10-100_ms":99,"lbm_reads_gt_100_ms":14,"lbm_reads_lt_1ms":14869,"lbm_write_time_us":1347409,"lbm_writes_1-10_ms":253,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":11341,"num_input_rowsets":5,"rows_written":6547831,"spinlock_wait_cycles":109205120,"thread_start_us":789,"threads_started":12} I0107 12:40:16.264336 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 136367566 bytes on disk I0107 12:40:16.457639 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.193s user 0.001s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":193089,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":7,"lbm_reads_lt_1ms":8} I0107 12:40:17.330394 27417 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=647871, attempt_no=0}) took 1012ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":16,"cfile_cache_hit":799,"cfile_cache_hit_bytes":3461742,"cfile_cache_miss":2454,"cfile_cache_miss_bytes":10364736,"lbm_read_time_us":925823,"lbm_reads_1-10_ms":24,"lbm_reads_10-100_ms":15,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":2413,"num_ops":7125,"prepare.queue_time_us":31,"prepare.run_cpu_time_us":8064,"prepare.run_wall_time_us":21283,"raft.queue_time_us":28,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":130,"replication_time_us":1955,"spinlock_wait_cycles":69120}]]} I0107 12:40:19.977725 24384 linked_list-test-util.h:519] Written 11000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 220000 Mean: 1897.97 Percentiles: 0% (min) = 778 25% = 1406 50% (med) = 1579 75% = 1813 95% = 2480 99% = 7752 99.9% = 43872 99.99% = 161792 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 230000 I0107 12:40:40.370255 24384 linked_list-test-util.h:519] Written 11500000 rows in chain Mean: 1896.05 Percentiles: 0% (min) = 778 25% = 1406 50% (med) = 1579 75% = 1812 95% = 2470 99% = 7760 99.9% = 43904 99.99% = 152192 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:40:51.875247 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033346 I0107 12:40:53.728910 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 1.854s user 0.637s sys 0.046s Metrics: {"bytes_written":7977130,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":7674,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":76939,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":65007,"lbm_writes_1-10_ms":14,"lbm_writes_lt_1ms":801,"reinsert_count":0,"rows_written":450456,"thread_start_us":534,"threads_started":1,"update_count":4261} I0107 12:40:53.729012 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33693720 bytes of WAL I0107 12:40:53.729056 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:40:53.729065 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000015 (ops 173963-186557) I0107 12:40:53.729092 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000016 (ops 186558-198904) I0107 12:40:53.729113 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000017 (ops 198905-211308) I0107 12:40:53.729131 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000018 (ops 211309-223741) I0107 12:40:53.741390 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.012s user 0.000s sys 0.010s Metrics: {} I0107 12:40:53.741581 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001466 I0107 12:40:54.174504 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033388 I0107 12:40:54.488160 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033345 I0107 12:40:56.305160 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.130s user 0.654s sys 0.046s Metrics: {"bytes_written":8010606,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":16736,"dirs.run_cpu_time_us":128,"dirs.run_wall_time_us":64335,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":98727,"lbm_writes_1-10_ms":15,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":804,"reinsert_count":0,"rows_written":454405,"thread_start_us":52,"threads_started":1,"update_count":1012} I0107 12:40:56.305559 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33595619 bytes of WAL I0107 12:40:56.305634 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:40:56.305699 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000015 (ops 173484-185900) I0107 12:40:56.305783 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000016 (ops 185901-198312) I0107 12:40:56.305852 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000017 (ops 198313-210718) I0107 12:40:56.305919 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000018 (ops 210719-223102) I0107 12:40:56.352560 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.047s user 0.000s sys 0.012s Metrics: {} I0107 12:40:56.356281 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 9026426 bytes on disk I0107 12:40:56.356834 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0107 12:40:56.356926 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001427 I0107 12:40:56.853607 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.365s user 0.686s sys 0.053s Metrics: {"bytes_written":8031926,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":85,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":133351,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":77947,"lbm_writes_1-10_ms":19,"lbm_writes_lt_1ms":802,"reinsert_count":0,"rows_written":454816,"thread_start_us":29,"threads_started":1,"update_count":2372} I0107 12:40:56.856730 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33598400 bytes of WAL I0107 12:40:56.856815 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:40:56.856866 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000015 (ops 173706-186113) I0107 12:40:56.857002 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000016 (ops 186114-198515) I0107 12:40:56.857069 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000017 (ops 198516-210901) I0107 12:40:56.857133 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000018 (ops 210902-223273) I0107 12:40:56.895370 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.039s user 0.000s sys 0.016s Metrics: {} I0107 12:40:56.896939 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001422 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 240000 Mean: 1898.7 Percentiles: 0% (min) = 778 25% = 1409 50% (med) = 1582 75% = 1815 95% = 2474 99% = 7804 99.9% = 43808 99.99% = 142592 100% (max) = 1068091 *NOTE: some values were greater than highest trackable value I0107 12:41:01.606904 24384 linked_list-test-util.h:519] Written 12000000 rows in chain I0107 12:41:04.466645 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 10.725s user 2.649s sys 0.261s Metrics: {"bytes_written":34729121,"cfile_cache_hit":93,"cfile_cache_hit_bytes":12267925,"cfile_cache_miss":4545,"cfile_cache_miss_bytes":128422063,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":599,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":144201,"drs_written":2,"lbm_read_time_us":1112726,"lbm_reads_1-10_ms":75,"lbm_reads_10-100_ms":21,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":4487,"lbm_write_time_us":582734,"lbm_writes_1-10_ms":71,"lbm_writes_10-100_ms":3,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3422,"num_input_rowsets":2,"rows_written":1964596,"spinlock_wait_cycles":45265920,"thread_start_us":486,"threads_started":9} I0107 12:41:04.469521 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 41262677 bytes on disk I0107 12:41:04.469633 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8} I0107 12:41:04.469691 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.033933 I0107 12:41:06.894654 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 10.537s user 2.409s sys 0.238s Metrics: {"bytes_written":34886534,"cfile_cache_hit":80,"cfile_cache_hit_bytes":10277495,"cfile_cache_miss":4576,"cfile_cache_miss_bytes":131064369,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":364,"dirs.run_wall_time_us":103027,"drs_written":2,"lbm_read_time_us":3096841,"lbm_reads_1-10_ms":52,"lbm_reads_10-100_ms":52,"lbm_reads_gt_100_ms":7,"lbm_reads_lt_1ms":4501,"lbm_write_time_us":337599,"lbm_writes_1-10_ms":59,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":3449,"num_input_rowsets":2,"rows_written":1973768,"spinlock_wait_cycles":25678720,"thread_start_us":516,"threads_started":8} I0107 12:41:06.894862 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.033605 I0107 12:41:07.242777 27676 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=714805, attempt_no=0}) took 1829ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1828713,"cfile_cache_hit":37,"cfile_cache_hit_bytes":188587,"num_ops":33,"prepare.queue_time_us":8,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":82,"raft.queue_time_us":102,"raft.run_cpu_time_us":33,"raft.run_wall_time_us":33,"replication_time_us":718}]]} I0107 12:41:07.667477 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.773s user 0.452s sys 0.016s Metrics: {"bytes_written":7944940,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":117027,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":139151,"lbm_writes_1-10_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":807,"rows_written":451377,"thread_start_us":36,"threads_started":1} I0107 12:41:07.667591 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41984156 bytes of WAL I0107 12:41:07.667634 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:41:07.667640 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000015 (ops 174151-186683) I0107 12:41:07.667666 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000016 (ops 186684-199066) I0107 12:41:07.667690 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000017 (ops 199067-211505) I0107 12:41:07.667706 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000018 (ops 211506-223917) I0107 12:41:07.667721 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000019 (ops 223918-236393) I0107 12:41:07.681015 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.013s user 0.002s sys 0.011s Metrics: {} I0107 12:41:07.681138 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.001457 I0107 12:41:08.038141 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 3.568s user 0.556s sys 0.017s Metrics: {"bytes_written":8139929,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":107368,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2844059,"lbm_writes_1-10_ms":4,"lbm_writes_10-100_ms":27,"lbm_writes_gt_100_ms":10,"lbm_writes_lt_1ms":794,"reinsert_count":0,"rows_written":461651,"thread_start_us":40,"threads_started":1,"update_count":1346} I0107 12:41:08.038266 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 42030239 bytes of WAL I0107 12:41:08.038327 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:41:08.038342 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000015 (ops 174078-186683) I0107 12:41:08.038375 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000016 (ops 186684-199075) I0107 12:41:08.038404 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000017 (ops 199076-211530) I0107 12:41:08.038426 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000018 (ops 211531-223925) I0107 12:41:08.038444 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000019 (ops 223926-236402) I0107 12:41:08.175210 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.137s user 0.000s sys 0.010s Metrics: {} I0107 12:41:08.175338 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 9185127 bytes on disk I0107 12:41:08.175470 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":26,"lbm_reads_lt_1ms":4} I0107 12:41:08.175531 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.001352 I0107 12:41:08.377014 27760 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=714812, attempt_no=0}) took 1133ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":442,"cfile_cache_miss":16,"cfile_cache_miss_bytes":77191,"cfile_init":1,"lbm_read_time_us":1131234,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":13,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":2,"num_ops":15,"prepare.queue_time_us":217,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":230,"raft.queue_time_us":22,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":70,"replication_time_us":956,"spinlock_wait_cycles":64128,"thread_start_us":123,"threads_started":2}]]} I0107 12:41:08.483156 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 11.586s user 2.374s sys 0.222s Metrics: {"bytes_written":34912668,"cfile_cache_hit":88,"cfile_cache_hit_bytes":11470832,"cfile_cache_miss":4570,"cfile_cache_miss_bytes":129978284,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":615,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":179769,"drs_written":2,"lbm_read_time_us":4265345,"lbm_reads_1-10_ms":54,"lbm_reads_10-100_ms":59,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":4487,"lbm_write_time_us":472033,"lbm_writes_1-10_ms":45,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":3466,"num_input_rowsets":2,"rows_written":1975128,"spinlock_wait_cycles":3801216,"thread_start_us":621,"threads_started":9} I0107 12:41:08.483351 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 41500713 bytes on disk I0107 12:41:08.508491 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.025s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":24945,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":6} I0107 12:41:08.508591 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034432 I0107 12:41:09.880525 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 1.372s user 0.490s sys 0.015s Metrics: {"bytes_written":7967463,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":79,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":87189,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":836324,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":25,"lbm_writes_lt_1ms":772,"reinsert_count":0,"rows_written":452574,"spinlock_wait_cycles":1920,"thread_start_us":41,"threads_started":1,"update_count":19} I0107 12:41:09.880652 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41968787 bytes of WAL I0107 12:41:09.880723 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:41:09.880738 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000015 (ops 173904-186233) I0107 12:41:09.880766 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000016 (ops 186234-198646) I0107 12:41:09.880795 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000017 (ops 198647-211012) I0107 12:41:09.880815 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000018 (ops 211013-223392) I0107 12:41:09.880833 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000019 (ops 223393-235867) I0107 12:41:09.890415 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.010s user 0.000s sys 0.009s Metrics: {} I0107 12:41:09.890586 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 9011795 bytes on disk I0107 12:41:09.890714 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0107 12:41:09.890779 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.001445 I0107 12:41:16.860672 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 8.685s user 2.306s sys 0.192s Metrics: {"bytes_written":35508281,"cfile_cache_hit":79,"cfile_cache_hit_bytes":9842475,"cfile_cache_miss":4665,"cfile_cache_miss_bytes":134197043,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":746,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":162592,"drs_written":2,"lbm_read_time_us":3501090,"lbm_reads_1-10_ms":51,"lbm_reads_10-100_ms":91,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4558,"lbm_write_time_us":349952,"lbm_writes_1-10_ms":28,"lbm_writes_10-100_ms":2,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3546,"num_input_rowsets":2,"rows_written":2009423,"spinlock_wait_cycles":5504,"thread_start_us":879,"threads_started":10} I0107 12:41:16.860877 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 42298699 bytes on disk I0107 12:41:17.264385 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.403s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":403343,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:41:18.064918 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 10.384s user 2.272s sys 0.228s Metrics: {"bytes_written":35514779,"cfile_cache_hit":92,"cfile_cache_hit_bytes":12325261,"cfile_cache_miss":4646,"cfile_cache_miss_bytes":131776539,"cfile_init":9,"delete_count":0,"delta_iterators_relevant":4,"dirs.queue_time_us":83,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":33799,"drs_written":2,"lbm_read_time_us":4008573,"lbm_reads_1-10_ms":41,"lbm_reads_10-100_ms":92,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":4546,"lbm_write_time_us":1110190,"lbm_writes_1-10_ms":31,"lbm_writes_10-100_ms":33,"lbm_writes_lt_1ms":3518,"num_input_rowsets":2,"reinsert_count":0,"rows_written":2009843,"spinlock_wait_cycles":768,"thread_start_us":386,"threads_started":9,"update_count":8} I0107 12:41:18.065131 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 42308656 bytes on disk I0107 12:41:18.065302 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8} I0107 12:41:18.065387 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 41467683 bytes on disk I0107 12:41:18.258083 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.193s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":192562,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":3} I0107 12:41:20.393260 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 10.502s user 2.430s sys 0.243s Metrics: {"bytes_written":35514961,"cfile_cache_hit":80,"cfile_cache_hit_bytes":10235303,"cfile_cache_miss":4662,"cfile_cache_miss_bytes":133859901,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":105,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":128862,"drs_written":2,"lbm_read_time_us":4181197,"lbm_reads_1-10_ms":59,"lbm_reads_10-100_ms":82,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":4551,"lbm_write_time_us":520763,"lbm_writes_1-10_ms":25,"lbm_writes_10-100_ms":11,"lbm_writes_lt_1ms":3542,"num_input_rowsets":2,"rows_written":2009862,"spinlock_wait_cycles":16902528,"thread_start_us":646,"threads_started":9} I0107 12:41:20.393362 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 42310157 bytes on disk I0107 12:41:20.445152 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.051s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":50439,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. I0107 12:41:32.784269 24384 linked_list-test-util.h:519] Written 12500000 rows in chain ------------------------------------------------------------ Count: 250000 Mean: 1940.63 Percentiles: 0% (min) = 778 25% = 1411 50% (med) = 1585 75% = 1819 95% = 2484 99% = 8048 99.9% = 47008 99.99% = 198400 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:41:34.520428 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033398 I0107 12:41:38.783521 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033374 I0107 12:41:39.710891 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033388 W0107 12:41:44.464268 24482 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.204s user 0.000s sys 0.000s I0107 12:41:44.465222 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 9.945s user 2.806s sys 0.212s Metrics: {"bytes_written":32592746,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4436,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":19777,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":370117,"lbm_writes_1-10_ms":66,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":3204,"reinsert_count":0,"rows_written":1846413,"thread_start_us":30,"threads_started":1,"update_count":9457} I0107 12:41:44.465602 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.997614 I0107 12:41:50.626217 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 11.843s user 2.986s sys 0.232s Metrics: {"bytes_written":33360317,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":20907,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":22184,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":463442,"lbm_writes_1-10_ms":108,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":3233,"reinsert_count":0,"rows_written":1888123,"thread_start_us":29,"threads_started":1,"update_count":13203} I0107 12:41:50.626402 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.997349 I0107 12:41:52.144345 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 12.433s user 3.036s sys 0.259s Metrics: {"bytes_written":33431563,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4363,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":13182,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431542,"lbm_writes_1-10_ms":102,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":3249,"reinsert_count":0,"rows_written":1893083,"thread_start_us":29,"threads_started":1,"update_count":11891} I0107 12:41:52.144562 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 75753096 bytes of WAL I0107 12:41:52.144695 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 9 log segments from log reader I0107 12:41:52.144740 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000034 (ops 195410-201196) I0107 12:41:52.144832 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000035 (ops 201197-207269) I0107 12:41:52.144908 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000036 (ops 207270-213246) I0107 12:41:52.144974 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000037 (ops 213247-219061) I0107 12:41:52.145074 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000038 (ops 219062-224983) I0107 12:41:52.149302 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000039 (ops 224984-230863) I0107 12:41:52.149386 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000040 (ops 230864-236747) I0107 12:41:52.149474 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000041 (ops 236748-242692) I0107 12:41:52.149540 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000042 (ops 242693-248621) I0107 12:41:52.272312 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.128s user 0.000s sys 0.027s Metrics: {} I0107 12:41:52.272500 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 37625950 bytes on disk I0107 12:41:52.272635 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0107 12:41:52.272758 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.997254 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 260000 Mean: 1945.32 Percentiles: 0% (min) = 778 25% = 1413 50% (med) = 1588 75% = 1824 95% = 2498 99% = 8136 99.9% = 47136 99.99% = 197376 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:41:54.763823 24384 linked_list-test-util.h:519] Written 13000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 270000 Mean: 1955.58 Percentiles: 0% (min) = 778 25% = 1417 50% (med) = 1594 75% = 1834 95% = 2528 99% = 8208 99.9% = 47200 99.99% = 197376 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:42:18.558684 24384 linked_list-test-util.h:519] Written 13500000 rows in chain I0107 12:42:34.875953 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 50.410s user 11.133s sys 1.177s Metrics: {"bytes_written":147634245,"cfile_cache_hit":374,"cfile_cache_hit_bytes":52348045,"cfile_cache_miss":19324,"cfile_cache_miss_bytes":547811967,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":8362,"dirs.run_cpu_time_us":638,"dirs.run_wall_time_us":426518,"drs_written":5,"lbm_read_time_us":8624296,"lbm_reads_1-10_ms":321,"lbm_reads_10-100_ms":152,"lbm_reads_gt_100_ms":18,"lbm_reads_lt_1ms":18921,"lbm_write_time_us":1856719,"lbm_writes_1-10_ms":299,"lbm_writes_10-100_ms":10,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":14505,"num_input_rowsets":5,"rows_written":8359667,"spinlock_wait_cycles":394948736,"thread_start_us":576,"threads_started":13} I0107 12:42:34.876108 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 75840833 bytes of WAL I0107 12:42:34.876174 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 9 log segments from log reader I0107 12:42:34.876194 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000034 (ops 195152-201053) I0107 12:42:34.880127 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000035 (ops 201054-207004) I0107 12:42:34.880221 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000036 (ops 207005-212891) I0107 12:42:34.880291 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000037 (ops 212892-218862) I0107 12:42:34.880359 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000038 (ops 218863-224788) I0107 12:42:34.880476 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000039 (ops 224789-230650) I0107 12:42:34.880553 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000040 (ops 230651-236607) I0107 12:42:34.880628 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000041 (ops 236608-242539) I0107 12:42:34.880731 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000042 (ops 242540-248504) I0107 12:42:34.965952 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.090s user 0.000s sys 0.023s Metrics: {} I0107 12:42:34.969779 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 176438165 bytes on disk I0107 12:42:35.327204 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.357s user 0.000s sys 0.002s Metrics: {"cfile_init":5,"lbm_read_time_us":357063,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":8,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":10} I0107 12:42:37.936105 28257 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=822159, attempt_no=0}) took 1026ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":356,"cfile_cache_hit":147926,"cfile_cache_hit_bytes":1460853582,"num_ops":17908,"prepare.queue_time_us":8,"prepare.run_cpu_time_us":19131,"prepare.run_wall_time_us":65734,"raft.queue_time_us":54,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":176,"replication_time_us":3195,"spinlock_wait_cycles":75818112}]]} I0107 12:42:41.769112 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 51.143s user 11.227s sys 1.159s Metrics: {"bytes_written":148805632,"cfile_cache_hit":412,"cfile_cache_hit_bytes":57917144,"cfile_cache_miss":19442,"cfile_cache_miss_bytes":547122516,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":28963,"dirs.run_cpu_time_us":607,"dirs.run_wall_time_us":580984,"drs_written":5,"lbm_read_time_us":8928195,"lbm_reads_1-10_ms":342,"lbm_reads_10-100_ms":140,"lbm_reads_gt_100_ms":19,"lbm_reads_lt_1ms":19029,"lbm_write_time_us":1851438,"lbm_writes_1-10_ms":319,"lbm_writes_10-100_ms":14,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":14599,"num_input_rowsets":5,"rows_written":8426796,"spinlock_wait_cycles":256393600,"thread_start_us":213,"threads_started":5} I0107 12:42:41.769389 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 75624604 bytes of WAL I0107 12:42:41.769482 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 9 log segments from log reader I0107 12:42:41.769497 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000034 (ops 195045-200993) I0107 12:42:41.769534 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000035 (ops 200994-206836) I0107 12:42:41.769563 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000036 (ops 206837-212801) I0107 12:42:41.769587 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000037 (ops 212802-218687) I0107 12:42:41.769608 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000038 (ops 218688-224626) I0107 12:42:41.769627 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000039 (ops 224627-230536) I0107 12:42:41.769649 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000040 (ops 230537-236411) I0107 12:42:41.769670 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000041 (ops 236412-242336) I0107 12:42:41.769690 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000042 (ops 242337-248187) I0107 12:42:41.810076 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.041s user 0.000s sys 0.022s Metrics: {} I0107 12:42:41.816807 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 177945122 bytes on disk I0107 12:42:42.037894 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.221s user 0.001s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":220736,"lbm_reads_1-10_ms":6,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":10} I0107 12:42:42.540052 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 50.267s user 11.314s sys 1.160s Metrics: {"bytes_written":149053988,"cfile_cache_hit":411,"cfile_cache_hit_bytes":58025514,"cfile_cache_miss":19473,"cfile_cache_miss_bytes":548038342,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":15908,"dirs.run_cpu_time_us":566,"dirs.run_wall_time_us":342644,"drs_written":5,"lbm_read_time_us":8985189,"lbm_reads_1-10_ms":334,"lbm_reads_10-100_ms":157,"lbm_reads_gt_100_ms":17,"lbm_reads_lt_1ms":19049,"lbm_write_time_us":1737893,"lbm_writes_1-10_ms":322,"lbm_writes_10-100_ms":21,"lbm_writes_lt_1ms":14616,"num_input_rowsets":5,"rows_written":8440914,"spinlock_wait_cycles":354185472,"thread_start_us":615,"threads_started":13} I0107 12:42:42.540230 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 178264107 bytes on disk I0107 12:42:42.540429 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":30,"lbm_reads_lt_1ms":20} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 280000 Mean: 1973.7 Percentiles: 0% (min) = 778 25% = 1421 50% (med) = 1600 75% = 1843 95% = 2550 99% = 8296 99.9% = 48192 99.99% = 215296 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:42:44.462538 24384 linked_list-test-util.h:519] Written 14000000 rows in chain I0107 12:42:44.978540 24384 linked_list-test-util.h:528] Sample at HT timestamp: 6465248317346463744 Inserted count: 14012700 I0107 12:42:53.812577 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033357 I0107 12:42:55.372051 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 1.559s user 0.556s sys 0.047s Metrics: {"bytes_written":7217452,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":7665,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":92894,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":65296,"lbm_writes_1-10_ms":14,"lbm_writes_lt_1ms":722,"reinsert_count":0,"rows_written":405400,"thread_start_us":31,"threads_started":1,"update_count":1624} I0107 12:42:55.372221 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33645702 bytes of WAL I0107 12:42:55.373606 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:42:55.373661 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000019 (ops 223742-236187) I0107 12:42:55.373764 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000020 (ops 236188-248576) I0107 12:42:55.373989 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000021 (ops 248577-261092) I0107 12:42:55.374052 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000022 (ops 261093-273504) I0107 12:42:55.405575 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.032s user 0.001s sys 0.010s Metrics: {} I0107 12:42:55.405755 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.010243 I0107 12:42:56.345369 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033345 I0107 12:42:57.055222 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.033389 I0107 12:42:58.515743 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.170s user 0.593s sys 0.050s Metrics: {"bytes_written":7245151,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":23673,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":50115,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":80528,"lbm_writes_1-10_ms":17,"lbm_writes_lt_1ms":721,"reinsert_count":0,"rows_written":405931,"thread_start_us":32,"threads_started":1,"update_count":3528} I0107 12:42:58.519207 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33732502 bytes of WAL I0107 12:42:58.519320 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:42:58.519369 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000019 (ops 223103-235735) I0107 12:42:58.519485 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000020 (ops 235736-248111) I0107 12:42:58.519629 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000021 (ops 248112-260497) I0107 12:42:58.519691 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000022 (ops 260498-272892) I0107 12:42:58.569159 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.050s user 0.000s sys 0.011s Metrics: {} I0107 12:42:58.569389 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 8036590 bytes on disk I0107 12:42:58.570185 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0107 12:42:58.570663 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.010141 I0107 12:42:59.041523 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 1.986s user 0.574s sys 0.036s Metrics: {"bytes_written":7253518,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":64965,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":89409,"lbm_writes_1-10_ms":18,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":721,"reinsert_count":0,"rows_written":407180,"spinlock_wait_cycles":1495808,"thread_start_us":30,"threads_started":1,"update_count":2125} I0107 12:42:59.041719 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33583546 bytes of WAL I0107 12:42:59.041807 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:42:59.041818 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000019 (ops 223274-235735) I0107 12:42:59.041846 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000020 (ops 235736-248110) I0107 12:42:59.041865 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000021 (ops 248111-260497) I0107 12:42:59.041882 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000022 (ops 260498-272892) I0107 12:42:59.103746 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.062s user 0.000s sys 0.012s Metrics: {} I0107 12:42:59.103899 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.010114 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 290000 Mean: 1978.56 Percentiles: 0% (min) = 778 25% = 1423 50% (med) = 1603 75% = 1847 95% = 2560 99% = 8384 99.9% = 48224 99.99% = 214016 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:43:07.074414 24384 linked_list-test-util.h:519] Written 14500000 rows in chain I0107 12:43:09.374042 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 13.968s user 3.103s sys 0.370s Metrics: {"bytes_written":41789214,"cfile_cache_hit":102,"cfile_cache_hit_bytes":12626561,"cfile_cache_miss":5494,"cfile_cache_miss_bytes":158429417,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":8369,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":167713,"drs_written":2,"lbm_read_time_us":1727677,"lbm_reads_1-10_ms":101,"lbm_reads_10-100_ms":53,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":5390,"lbm_write_time_us":558697,"lbm_writes_1-10_ms":118,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":4085,"num_input_rowsets":3,"rows_written":2369996,"spinlock_wait_cycles":52933760,"thread_start_us":1300,"threads_started":10} I0107 12:43:09.374217 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.033704 I0107 12:43:11.505036 28428 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=862318, attempt_no=0}) took 1052ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1051768,"cfile_cache_hit":41,"cfile_cache_hit_bytes":216743,"num_ops":36,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":45,"raft.run_cpu_time_us":26,"raft.run_wall_time_us":26,"replication_time_us":291}]]} I0107 12:43:12.548801 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 13.978s user 3.061s sys 0.307s Metrics: {"bytes_written":41961219,"cfile_cache_hit":103,"cfile_cache_hit_bytes":12922043,"cfile_cache_miss":5521,"cfile_cache_miss_bytes":158866773,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":2349,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":185169,"drs_written":2,"lbm_read_time_us":3561542,"lbm_reads_1-10_ms":90,"lbm_reads_10-100_ms":62,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":5409,"lbm_write_time_us":408833,"lbm_writes_1-10_ms":91,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":4132,"num_input_rowsets":3,"rows_written":2379699,"spinlock_wait_cycles":76011264,"thread_start_us":75,"threads_started":2} I0107 12:43:12.548976 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034689 I0107 12:43:12.717799 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 3.343s user 0.605s sys 0.033s Metrics: {"bytes_written":7390410,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1866,"dirs.run_cpu_time_us":51,"dirs.run_wall_time_us":51,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2040419,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":32,"lbm_writes_gt_100_ms":4,"lbm_writes_lt_1ms":708,"reinsert_count":0,"rows_written":414760,"thread_start_us":31,"threads_started":1,"update_count":2665} I0107 12:43:12.717977 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 25192358 bytes of WAL I0107 12:43:12.718232 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 3 log segments from log reader I0107 12:43:12.718253 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000020 (ops 236403-248797) I0107 12:43:12.718288 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000021 (ops 248798-261261) I0107 12:43:12.718317 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000022 (ops 261262-273706) I0107 12:43:12.733593 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.015s user 0.000s sys 0.009s Metrics: {} I0107 12:43:12.733786 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.009683 I0107 12:43:13.317139 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 14.213s user 3.091s sys 0.307s Metrics: {"bytes_written":42006805,"cfile_cache_hit":92,"cfile_cache_hit_bytes":11106094,"cfile_cache_miss":5538,"cfile_cache_miss_bytes":160878704,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":16982,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":151706,"drs_written":2,"lbm_read_time_us":3428084,"lbm_reads_1-10_ms":99,"lbm_reads_10-100_ms":72,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":5415,"lbm_write_time_us":753874,"lbm_writes_1-10_ms":83,"lbm_writes_10-100_ms":6,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":4139,"num_input_rowsets":3,"rows_written":2382308,"spinlock_wait_cycles":62206336,"thread_start_us":1655,"threads_started":10} I0107 12:43:13.317355 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.034288 I0107 12:43:13.694914 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 1.146s user 0.495s sys 0.027s Metrics: {"bytes_written":7430042,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":5331,"dirs.run_cpu_time_us":46,"dirs.run_wall_time_us":46,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":319859,"lbm_writes_1-10_ms":36,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":718,"reinsert_count":0,"rows_written":418785,"thread_start_us":30,"threads_started":1,"update_count":27} I0107 12:43:13.695027 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 25214342 bytes of WAL I0107 12:43:13.695097 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 3 log segments from log reader I0107 12:43:13.695112 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000020 (ops 236394-248786) I0107 12:43:13.695143 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000021 (ops 248787-261261) I0107 12:43:13.695178 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000022 (ops 261262-273686) I0107 12:43:13.701079 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.006s user 0.000s sys 0.006s Metrics: {} I0107 12:43:13.701172 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 50515971 bytes on disk I0107 12:43:13.795502 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.094s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":94133,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":4} I0107 12:43:13.795639 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.009640 I0107 12:43:15.442705 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 2.125s user 0.579s sys 0.041s Metrics: {"bytes_written":7472835,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":658,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":72793,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612756,"lbm_writes_1-10_ms":24,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":726,"reinsert_count":0,"rows_written":420224,"thread_start_us":30,"threads_started":1,"update_count":1485} I0107 12:43:15.442884 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 33608196 bytes of WAL I0107 12:43:15.442987 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 4 log segments from log reader I0107 12:43:15.443033 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000020 (ops 235868-248240) I0107 12:43:15.443099 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000021 (ops 248241-260637) I0107 12:43:15.443199 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000022 (ops 260638-273061) I0107 12:43:15.443315 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000023 (ops 273062-285458) I0107 12:43:15.466118 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.023s user 0.000s sys 0.011s Metrics: {} I0107 12:43:15.466269 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 50578936 bytes on disk I0107 12:43:15.492576 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.026s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":26141,"lbm_reads_1-10_ms":4,"lbm_reads_lt_1ms":4} I0107 12:43:15.492736 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.009624 I0107 12:43:25.607795 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 12.874s user 3.092s sys 0.353s Metrics: {"bytes_written":42733277,"cfile_cache_hit":98,"cfile_cache_hit_bytes":11901110,"cfile_cache_miss":5628,"cfile_cache_miss_bytes":163229544,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":1447,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":79035,"drs_written":2,"lbm_read_time_us":3115199,"lbm_reads_1-10_ms":84,"lbm_reads_10-100_ms":76,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":5516,"lbm_write_time_us":514359,"lbm_writes_1-10_ms":66,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":4236,"num_input_rowsets":3,"rows_written":2424183,"spinlock_wait_cycles":45860352,"thread_start_us":558,"threads_started":10} I0107 12:43:25.608011 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 51550570 bytes on disk I0107 12:43:25.894815 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.287s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":286606,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:43:25.894973 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 50295194 bytes on disk I0107 12:43:26.019474 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.124s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":124319,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":4} I0107 12:43:27.737967 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 13.942s user 3.086s sys 0.321s Metrics: {"bytes_written":42808766,"cfile_cache_hit":104,"cfile_cache_hit_bytes":12688861,"cfile_cache_miss":5638,"cfile_cache_miss_bytes":162747105,"cfile_init":14,"delta_iterators_relevant":10,"dirs.queue_time_us":3664,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":184098,"drs_written":2,"lbm_read_time_us":3685230,"lbm_reads_1-10_ms":80,"lbm_reads_10-100_ms":86,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":5524,"lbm_write_time_us":624017,"lbm_writes_1-10_ms":67,"lbm_writes_10-100_ms":6,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":4242,"num_input_rowsets":3,"rows_written":2428628,"spinlock_wait_cycles":22223232,"thread_start_us":71,"threads_started":2} I0107 12:43:27.738090 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 51650814 bytes on disk I0107 12:43:27.738214 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} I0107 12:43:28.773923 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 13.281s user 3.097s sys 0.342s Metrics: {"bytes_written":42840892,"cfile_cache_hit":106,"cfile_cache_hit_bytes":13152378,"cfile_cache_miss":5632,"cfile_cache_miss_bytes":162398134,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":166666,"drs_written":2,"lbm_read_time_us":3965559,"lbm_reads_1-10_ms":75,"lbm_reads_10-100_ms":72,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":5526,"lbm_write_time_us":443082,"lbm_writes_1-10_ms":70,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":4243,"num_input_rowsets":3,"rows_written":2430086,"spinlock_wait_cycles":27668224,"thread_start_us":511,"threads_started":10} I0107 12:43:28.774027 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 51687818 bytes on disk I0107 12:43:28.774139 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 300000 Mean: 2000.57 Percentiles: 0% (min) = 778 25% = 1425 50% (med) = 1606 75% = 1852 95% = 2576 99% = 8568 99.9% = 49280 99.99% = 222208 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:43:35.123307 24384 linked_list-test-util.h:519] Written 15000000 rows in chain I0107 12:43:44.501770 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033343 I0107 12:43:50.418568 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 5.917s user 2.445s sys 0.169s Metrics: {"bytes_written":31695974,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":54965,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":241101,"lbm_writes_1-10_ms":19,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":3139,"reinsert_count":0,"rows_written":1782486,"thread_start_us":37,"threads_started":1,"update_count":8714} I0107 12:43:50.418764 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 75758688 bytes of WAL I0107 12:43:50.418831 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 9 log segments from log reader I0107 12:43:50.418900 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000043 (ops 248505-254423) I0107 12:43:50.418969 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000044 (ops 254424-260326) I0107 12:43:50.419034 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000045 (ops 260327-266233) I0107 12:43:50.419092 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000046 (ops 266234-272135) I0107 12:43:50.419171 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000047 (ops 272136-278043) I0107 12:43:50.419230 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000048 (ops 278044-283911) I0107 12:43:50.419286 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000049 (ops 283912-289933) I0107 12:43:50.419339 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000050 (ops 289934-295780) I0107 12:43:50.419394 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000051 (ops 295781-301736) I0107 12:43:50.464852 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.046s user 0.000s sys 0.025s Metrics: {} I0107 12:43:50.465039 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 35583528 bytes on disk I0107 12:43:50.465296 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4} I0107 12:43:50.465384 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.770507 I0107 12:43:50.805128 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033383 I0107 12:43:52.301745 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033377 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 310000 Mean: 2004.11 Percentiles: 0% (min) = 778 25% = 1427 50% (med) = 1609 75% = 1856 95% = 2582 99% = 8640 99.9% = 49280 99.99% = 222208 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:43:58.016724 24384 linked_list-test-util.h:519] Written 15500000 rows in chain I0107 12:44:02.289510 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 11.484s user 2.899s sys 0.219s Metrics: {"bytes_written":32088576,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":134,"dirs.run_wall_time_us":201952,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":348831,"lbm_writes_1-10_ms":63,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":3131,"reinsert_count":0,"rows_written":1803138,"thread_start_us":37,"threads_started":1,"update_count":11830} I0107 12:44:02.289755 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 84210083 bytes of WAL I0107 12:44:02.289858 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 10 log segments from log reader I0107 12:44:02.289904 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000043 (ops 248188-254114) I0107 12:44:02.290004 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000044 (ops 254115-259997) I0107 12:44:02.290094 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000045 (ops 259998-265865) I0107 12:44:02.290158 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000046 (ops 265866-271963) I0107 12:44:02.290237 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000047 (ops 271964-277894) I0107 12:44:02.290345 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000048 (ops 277895-283784) I0107 12:44:02.290450 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000049 (ops 283785-289603) I0107 12:44:02.290516 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000050 (ops 289604-295613) I0107 12:44:02.290587 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000051 (ops 295614-301453) I0107 12:44:02.290652 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000052 (ops 301454-307379) I0107 12:44:02.353674 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.064s user 0.001s sys 0.026s Metrics: {} I0107 12:44:02.353817 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 35984069 bytes on disk I0107 12:44:02.353950 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0107 12:44:02.354058 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.771495 W0107 12:44:02.625452 24607 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.207s user 0.001s sys 0.000s I0107 12:44:02.626947 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 10.325s user 2.746s sys 0.201s Metrics: {"bytes_written":32131643,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":786,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":30842,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":330710,"lbm_writes_1-10_ms":58,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3151,"reinsert_count":0,"rows_written":1810775,"spinlock_wait_cycles":30464,"thread_start_us":31,"threads_started":1,"update_count":3504} I0107 12:44:02.627113 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.771555 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 320000 Mean: 2012.18 Percentiles: 0% (min) = 778 25% = 1430 50% (med) = 1613 75% = 1863 95% = 2600 99% = 8720 99.9% = 49856 99.99% = 222208 100% (max) = 1830262 *NOTE: some values were greater than highest trackable value I0107 12:44:21.960701 24384 linked_list-test-util.h:519] Written 16000000 rows in chain I0107 12:44:40.798926 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 50.333s user 10.898s sys 1.160s Metrics: {"bytes_written":145164469,"cfile_cache_hit":295,"cfile_cache_hit_bytes":39489238,"cfile_cache_miss":19119,"cfile_cache_miss_bytes":555776516,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":323,"dirs.run_cpu_time_us":543,"dirs.run_wall_time_us":1180613,"drs_written":5,"lbm_read_time_us":7951216,"lbm_reads_1-10_ms":356,"lbm_reads_10-100_ms":146,"lbm_reads_gt_100_ms":17,"lbm_reads_lt_1ms":18684,"lbm_write_time_us":2171836,"lbm_writes_1-10_ms":297,"lbm_writes_10-100_ms":14,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":14286,"num_input_rowsets":5,"rows_written":8237132,"spinlock_wait_cycles":347361408,"thread_start_us":164,"threads_started":5} I0107 12:44:40.799140 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.303878 W0107 12:44:43.643126 28992 lock_manager.cc:368] Waited 1 seconds to obtain row lock on key \xb0\x207\x08\xab\xce\x00\x04 cur holder: 0x508ee00 I0107 12:44:44.167718 28927 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=972656, attempt_no=0}) took 2449ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":2448313,"cfile_cache_hit":10,"cfile_cache_hit_bytes":52314,"num_ops":8,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":117,"raft.run_cpu_time_us":41,"raft.run_wall_time_us":504,"replication_time_us":417,"thread_start_us":35,"threads_started":1}]]} I0107 12:44:44.169253 28927 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=972661, attempt_no=0}) took 1527ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1524272,"cfile_cache_hit":281,"cfile_cache_hit_bytes":1153786,"num_ops":311,"prepare.queue_time_us":234,"prepare.run_cpu_time_us":465,"prepare.run_wall_time_us":465,"raft.queue_time_us":113,"raft.run_cpu_time_us":39,"raft.run_wall_time_us":78,"replication_time_us":825,"spinlock_wait_cycles":4480,"thread_start_us":43,"threads_started":2}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 330000 Mean: 2026.77 Percentiles: 0% (min) = 778 25% = 1433 50% (med) = 1617 75% = 1869 95% = 2616 99% = 8776 99.9% = 49984 99.99% = 230656 100% (max) = 2449511 *NOTE: some values were greater than highest trackable value I0107 12:44:48.161883 24384 linked_list-test-util.h:519] Written 16500000 rows in chain I0107 12:44:54.051817 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 51.698s user 10.957s sys 1.117s Metrics: {"bytes_written":146664284,"cfile_cache_hit":330,"cfile_cache_hit_bytes":45339710,"cfile_cache_miss":19284,"cfile_cache_miss_bytes":556282558,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":5293,"dirs.run_cpu_time_us":661,"dirs.run_wall_time_us":731590,"drs_written":5,"lbm_read_time_us":10102440,"lbm_reads_1-10_ms":394,"lbm_reads_10-100_ms":171,"lbm_reads_gt_100_ms":20,"lbm_reads_lt_1ms":18783,"lbm_write_time_us":1623640,"lbm_writes_1-10_ms":281,"lbm_writes_10-100_ms":10,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14459,"num_input_rowsets":5,"rows_written":8323208,"spinlock_wait_cycles":495699840,"thread_start_us":172,"threads_started":5} I0107 12:44:54.052013 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.299331 I0107 12:44:55.013001 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 52.386s user 10.871s sys 1.116s Metrics: {"bytes_written":147016895,"cfile_cache_hit":347,"cfile_cache_hit_bytes":47764228,"cfile_cache_miss":19313,"cfile_cache_miss_bytes":555312834,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":2402,"dirs.run_cpu_time_us":556,"dirs.run_wall_time_us":344418,"drs_written":5,"lbm_read_time_us":10694863,"lbm_reads_1-10_ms":311,"lbm_reads_10-100_ms":186,"lbm_reads_gt_100_ms":23,"lbm_reads_lt_1ms":18881,"lbm_write_time_us":1804731,"lbm_writes_1-10_ms":357,"lbm_writes_10-100_ms":15,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14413,"num_input_rowsets":5,"rows_written":8343311,"spinlock_wait_cycles":458694912,"thread_start_us":693,"threads_started":13} I0107 12:44:55.013214 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 84069064 bytes of WAL I0107 12:44:55.013293 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 10 log segments from log reader I0107 12:44:55.013303 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000043 (ops 248622-254561) I0107 12:44:55.013335 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000044 (ops 254562-260460) I0107 12:44:55.013356 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000045 (ops 260461-266374) I0107 12:44:55.013375 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000046 (ops 266375-272298) I0107 12:44:55.013392 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000047 (ops 272299-278219) I0107 12:44:55.013411 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000048 (ops 278220-284140) I0107 12:44:55.013453 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000049 (ops 284141-290052) I0107 12:44:55.013473 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000050 (ops 290053-296008) I0107 12:44:55.013490 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000051 (ops 296009-301926) I0107 12:44:55.013530 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000052 (ops 301927-307855) I0107 12:44:55.037043 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.024s user 0.001s sys 0.022s Metrics: {} I0107 12:44:55.037169 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 177722970 bytes on disk I0107 12:44:55.416267 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.379s user 0.001s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":378738,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":9,"lbm_reads_lt_1ms":9} I0107 12:44:55.416421 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.300130 I0107 12:44:56.263011 29033 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=985549, attempt_no=0}) took 1932ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":31,"cfile_cache_hit":1477,"cfile_cache_hit_bytes":6266737,"cfile_cache_miss":3574,"cfile_cache_miss_bytes":14816033,"lbm_read_time_us":1825940,"lbm_reads_1-10_ms":51,"lbm_reads_10-100_ms":41,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":3479,"num_ops":10703,"prepare.queue_time_us":11,"prepare.run_cpu_time_us":11610,"prepare.run_wall_time_us":48721,"raft.queue_time_us":80,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"replication_time_us":2004,"spinlock_wait_cycles":768}]]} W0107 12:44:58.638201 29126 lock_manager.cc:368] Waited 1 seconds to obtain row lock on key \xf95\x1b\xcb\xb3\xe4\x00\x00 cur holder: 0x7d6f180 W0107 12:44:59.638387 29126 lock_manager.cc:368] Waited 2 seconds to obtain row lock on key \xf95\x1b\xcb\xb3\xe4\x00\x00 cur holder: 0x7d6f180 I0107 12:45:00.191262 29017 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=986219, attempt_no=0}) took 3087ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":3085874,"cfile_cache_hit":10,"cfile_cache_hit_bytes":52743,"num_ops":8,"prepare.queue_time_us":18,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"replication_time_us":1117,"thread_start_us":36,"threads_started":1,"wal-append.queue_time_us":55}]]} I0107 12:45:02.154489 29096 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=986223, attempt_no=0}) took 1962ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1961550,"cfile_cache_hit":8,"cfile_cache_hit_bytes":44681,"num_ops":7,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":406,"raft.queue_time_us":31,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":72,"replication_time_us":845,"spinlock_wait_cycles":67584,"thread_start_us":23,"threads_started":1}]]} I0107 12:45:02.166023 29095 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=986221, attempt_no=0}) took 4533ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":4519604,"cfile_cache_hit":1015,"cfile_cache_hit_bytes":4320135,"num_ops":1771,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":1841,"prepare.run_wall_time_us":2136,"raft.queue_time_us":25,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":85,"replication_time_us":604,"spinlock_wait_cycles":671488,"thread_start_us":30,"threads_started":1}]]} I0107 12:45:14.466717 24901 thread.cc:387] Could not get per-thread statistics: Not found: /proc/self/task/29257/stat: No such file or directory (error 2) I0107 12:45:18.559594 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 24.507s user 4.577s sys 0.445s Metrics: {"bytes_written":67455899,"cfile_cache_hit":95,"cfile_cache_hit_bytes":12048177,"cfile_cache_miss":8921,"cfile_cache_miss_bytes":266201855,"cfile_init":9,"delta_iterators_relevant":4,"dirs.queue_time_us":11692,"dirs.run_cpu_time_us":195,"dirs.run_wall_time_us":241755,"drs_written":2,"lbm_read_time_us":10760249,"lbm_reads_1-10_ms":146,"lbm_reads_10-100_ms":190,"lbm_reads_gt_100_ms":16,"lbm_reads_lt_1ms":8605,"lbm_write_time_us":1300988,"lbm_writes_1-10_ms":74,"lbm_writes_10-100_ms":22,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":6682,"num_input_rowsets":2,"rows_written":3828789,"spinlock_wait_cycles":32216704,"thread_start_us":136,"threads_started":2} I0107 12:45:18.560184 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 218548916 bytes on disk I0107 12:45:18.792212 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.231s user 0.000s sys 0.001s Metrics: {"cfile_init":6,"lbm_read_time_us":231085,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":12,"lbm_reads_lt_1ms":11} I0107 12:45:18.792340 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.039839 I0107 12:45:20.852308 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 2.060s user 0.620s sys 0.040s Metrics: {"bytes_written":8046044,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":43385,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":701684,"lbm_writes_1-10_ms":16,"lbm_writes_10-100_ms":18,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":788,"reinsert_count":0,"rows_written":455172,"thread_start_us":74,"threads_started":1,"update_count":2415} I0107 12:45:20.852504 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33593579 bytes of WAL I0107 12:45:20.852931 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:45:20.853667 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000023 (ops 273505-285888) I0107 12:45:20.853775 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000024 (ops 285889-298291) I0107 12:45:20.853839 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000025 (ops 298292-310721) I0107 12:45:20.853899 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000026 (ops 310722-323163) I0107 12:45:20.872490 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.020s user 0.000s sys 0.011s Metrics: {} I0107 12:45:20.872632 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 9034658 bytes on disk I0107 12:45:20.873800 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":370,"lbm_reads_lt_1ms":4} I0107 12:45:20.873889 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.005823 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 340000 Mean: 2065.71 Percentiles: 0% (min) = 778 25% = 1436 50% (med) = 1621 75% = 1875 95% = 2632 99% = 8872 99.9% = 51712 99.99% = 266496 100% (max) = 3087444 *NOTE: some values were greater than highest trackable value I0107 12:45:22.950517 24384 linked_list-test-util.h:519] Written 17000000 rows in chain I0107 12:45:32.996150 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 52.197s user 9.867s sys 1.012s Metrics: {"bytes_written":144619517,"cfile_cache_hit":163,"cfile_cache_hit_bytes":19737296,"cfile_cache_miss":19179,"cfile_cache_miss_bytes":578677482,"cfile_init":24,"delta_iterators_relevant":10,"dirs.queue_time_us":4054,"dirs.run_cpu_time_us":629,"dirs.run_wall_time_us":440966,"drs_written":5,"lbm_read_time_us":18300766,"lbm_reads_1-10_ms":368,"lbm_reads_10-100_ms":357,"lbm_reads_gt_100_ms":30,"lbm_reads_lt_1ms":18520,"lbm_write_time_us":1738077,"lbm_writes_1-10_ms":257,"lbm_writes_10-100_ms":13,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14281,"num_input_rowsets":5,"rows_written":8208786,"spinlock_wait_cycles":194168960,"thread_start_us":245,"threads_started":5} I0107 12:45:32.996366 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 216556091 bytes on disk I0107 12:45:33.003661 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"cfile_init":6,"lbm_read_time_us":7133,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":22} I0107 12:45:33.003768 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.042913 W0107 12:45:35.677531 29400 lock_manager.cc:368] Waited 1 seconds to obtain row lock on key \xcf\x89\x07\xe2\xdb\xb4\x00\x07 cur holder: 0x508ee00 I0107 12:45:35.794066 29329 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1020319, attempt_no=0}) took 2006ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":2004578,"cfile_cache_hit":9,"cfile_cache_hit_bytes":48208,"num_ops":7,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":1580,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":20,"replication_time_us":1881,"thread_start_us":40,"threads_started":1}]]} I0107 12:45:35.822204 29372 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1020321, attempt_no=0}) took 1146ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1141698,"cfile_cache_hit":422,"cfile_cache_hit_bytes":1743986,"num_ops":481,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":714,"raft.queue_time_us":89,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":50,"replication_time_us":1665,"spinlock_wait_cycles":1408,"thread_start_us":58,"threads_started":2}]]} I0107 12:45:36.850136 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 15.976s user 3.407s sys 0.337s Metrics: {"bytes_written":49728838,"cfile_cache_hit":113,"cfile_cache_hit_bytes":14354418,"cfile_cache_miss":6549,"cfile_cache_miss_bytes":190341904,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":148634,"drs_written":2,"lbm_read_time_us":6076845,"lbm_reads_1-10_ms":84,"lbm_reads_10-100_ms":111,"lbm_reads_gt_100_ms":12,"lbm_reads_lt_1ms":6390,"lbm_write_time_us":573765,"lbm_writes_1-10_ms":88,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":4920,"num_input_rowsets":3,"rows_written":2825168,"spinlock_wait_cycles":21298176,"thread_start_us":425,"threads_started":10} I0107 12:45:36.850318 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.040037 I0107 12:45:37.270110 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 4.266s user 0.648s sys 0.051s Metrics: {"bytes_written":8615400,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":128,"dirs.run_cpu_time_us":71,"dirs.run_wall_time_us":72,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2741018,"lbm_writes_1-10_ms":32,"lbm_writes_10-100_ms":57,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":789,"reinsert_count":0,"rows_written":488439,"thread_start_us":76,"threads_started":1,"update_count":1773} I0107 12:45:37.270226 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 41980686 bytes of WAL I0107 12:45:37.270296 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:45:37.270310 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000023 (ops 272893-285290) I0107 12:45:37.270349 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000024 (ops 285291-297703) I0107 12:45:37.270382 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000025 (ops 297704-310109) I0107 12:45:37.270428 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000026 (ops 310110-322565) I0107 12:45:37.270457 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000027 (ops 322566-334944) I0107 12:45:37.280375 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.010s user 0.000s sys 0.009s Metrics: {} I0107 12:45:37.280503 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 9719587 bytes on disk I0107 12:45:37.323268 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.043s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42652,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3} I0107 12:45:37.323355 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.005384 I0107 12:45:40.102910 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 3.252s user 0.648s sys 0.050s Metrics: {"bytes_written":7911683,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6831,"dirs.run_cpu_time_us":53,"dirs.run_wall_time_us":52,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1146465,"lbm_writes_1-10_ms":46,"lbm_writes_10-100_ms":23,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":741,"reinsert_count":0,"rows_written":449125,"thread_start_us":43,"threads_started":1,"update_count":26} I0107 12:45:40.103029 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41971288 bytes of WAL I0107 12:45:40.103091 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:45:40.103107 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000023 (ops 273707-286129) I0107 12:45:40.103153 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000024 (ops 286130-298580) I0107 12:45:40.103179 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000025 (ops 298581-311053) I0107 12:45:40.103204 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000026 (ops 311054-323461) I0107 12:45:40.103221 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000027 (ops 323462-335924) I0107 12:45:40.113565 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.010s user 0.000s sys 0.011s Metrics: {} I0107 12:45:40.113682 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.005358 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 350000 Mean: 2081.27 Percentiles: 0% (min) = 778 25% = 1438 50% (med) = 1624 75% = 1880 95% = 2644 99% = 8984 99.9% = 52224 99.99% = 288768 100% (max) = 3087444 *NOTE: some values were greater than highest trackable value I0107 12:45:50.264905 24384 linked_list-test-util.h:519] Written 17500000 rows in chain W0107 12:45:50.565503 24607 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.234s user 0.000s sys 0.000s I0107 12:45:50.566644 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 55.150s user 10.416s sys 1.161s Metrics: {"bytes_written":146533740,"cfile_cache_hit":172,"cfile_cache_hit_bytes":20762133,"cfile_cache_miss":19426,"cfile_cache_miss_bytes":585863031,"cfile_init":20,"delta_iterators_relevant":10,"dirs.queue_time_us":14832,"dirs.run_cpu_time_us":720,"dirs.run_wall_time_us":440541,"drs_written":5,"lbm_read_time_us":16941086,"lbm_reads_1-10_ms":362,"lbm_reads_10-100_ms":328,"lbm_reads_gt_100_ms":30,"lbm_reads_lt_1ms":18786,"lbm_write_time_us":4960518,"lbm_writes_1-10_ms":235,"lbm_writes_10-100_ms":98,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14410,"num_input_rowsets":5,"rows_written":8318314,"spinlock_wait_cycles":211096192,"thread_start_us":207,"threads_started":5} I0107 12:45:50.567013 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 177741647 bytes on disk I0107 12:45:50.567487 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":30,"lbm_reads_lt_1ms":20} I0107 12:45:50.567539 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.047646 I0107 12:45:53.384737 29430 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1037957, attempt_no=0}) took 1915ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1911541,"cfile_cache_hit":11,"cfile_cache_hit_bytes":56849,"num_ops":9,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":54,"raft.queue_time_us":236,"raft.run_cpu_time_us":23,"raft.run_wall_time_us":23,"replication_time_us":3914,"thread_start_us":32,"threads_started":1}]]} I0107 12:45:53.393362 29430 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1037959, attempt_no=0}) took 1069ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1057342,"cfile_cache_hit":365,"cfile_cache_hit_bytes":1510373,"num_ops":418,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":638,"prepare.run_wall_time_us":793,"raft.queue_time_us":45,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":74,"replication_time_us":5289,"spinlock_wait_cycles":68224,"thread_start_us":30,"threads_started":1}]]} I0107 12:45:54.128039 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 16.804s user 3.581s sys 0.384s Metrics: {"bytes_written":50479387,"cfile_cache_hit":125,"cfile_cache_hit_bytes":16081138,"cfile_cache_miss":6635,"cfile_cache_miss_bytes":191703256,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":2332,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":148752,"drs_written":2,"lbm_read_time_us":4776888,"lbm_reads_1-10_ms":103,"lbm_reads_10-100_ms":93,"lbm_reads_gt_100_ms":7,"lbm_reads_lt_1ms":6480,"lbm_write_time_us":623161,"lbm_writes_1-10_ms":115,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":4967,"num_input_rowsets":3,"rows_written":2868138,"spinlock_wait_cycles":18627328,"thread_start_us":1461,"threads_started":10} I0107 12:45:54.128203 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 61527004 bytes on disk I0107 12:45:54.128324 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0107 12:45:54.128409 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.044565 I0107 12:45:54.734525 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 4.167s user 0.679s sys 0.038s Metrics: {"bytes_written":9438757,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":54,"dirs.run_wall_time_us":55,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2695780,"lbm_writes_1-10_ms":33,"lbm_writes_10-100_ms":45,"lbm_writes_gt_100_ms":4,"lbm_writes_lt_1ms":884,"reinsert_count":0,"rows_written":535448,"thread_start_us":29,"threads_started":1,"update_count":2364} I0107 12:45:54.734638 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 41986217 bytes of WAL I0107 12:45:54.734686 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 5 log segments from log reader I0107 12:45:54.734699 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000023 (ops 272893-285290) I0107 12:45:54.734728 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000024 (ops 285291-297703) I0107 12:45:54.734750 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000025 (ops 297704-310128) I0107 12:45:54.734771 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000026 (ops 310129-322594) I0107 12:45:54.734789 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000027 (ops 322595-334983) I0107 12:45:54.744735 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.010s user 0.000s sys 0.010s Metrics: {} I0107 12:45:54.744827 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.004881 I0107 12:45:56.442286 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 16.328s user 3.581s sys 0.376s Metrics: {"bytes_written":50567016,"cfile_cache_hit":87,"cfile_cache_hit_bytes":10320462,"cfile_cache_miss":6693,"cfile_cache_miss_bytes":198083808,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":4270,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":121195,"drs_written":2,"lbm_read_time_us":4806808,"lbm_reads_1-10_ms":101,"lbm_reads_10-100_ms":93,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":6543,"lbm_write_time_us":530418,"lbm_writes_1-10_ms":79,"lbm_writes_10-100_ms":7,"lbm_writes_lt_1ms":5010,"num_input_rowsets":3,"rows_written":2873308,"spinlock_wait_cycles":9511168,"thread_start_us":1383,"threads_started":10} I0107 12:45:56.442482 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 61687792 bytes on disk I0107 12:45:56.442620 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":22,"lbm_reads_lt_1ms":8} I0107 12:45:56.442737 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 60550528 bytes on disk I0107 12:45:56.598345 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.156s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":155430,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:45:56.751777 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 2.623s user 0.644s sys 0.057s Metrics: {"bytes_written":8715737,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":96,"dirs.run_cpu_time_us":45,"dirs.run_wall_time_us":45,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1043814,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":11,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":858,"reinsert_count":0,"rows_written":493352,"thread_start_us":63,"threads_started":1,"update_count":3227} I0107 12:45:56.751922 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 41979064 bytes of WAL I0107 12:45:56.751997 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:45:56.752019 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000023 (ops 273687-286116) I0107 12:45:56.752048 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000024 (ops 286117-298563) I0107 12:45:56.752084 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000025 (ops 298564-310998) I0107 12:45:56.752121 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000026 (ops 310999-323405) I0107 12:45:56.752148 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000027 (ops 323406-335833) I0107 12:45:56.762095 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.010s user 0.000s sys 0.009s Metrics: {} I0107 12:45:56.762233 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.004864 I0107 12:46:02.353327 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033351 I0107 12:46:10.865164 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 16.120s user 3.755s sys 0.405s Metrics: {"bytes_written":51344273,"cfile_cache_hit":114,"cfile_cache_hit_bytes":14501776,"cfile_cache_miss":6766,"cfile_cache_miss_bytes":196796356,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":21273,"dirs.run_cpu_time_us":321,"dirs.run_wall_time_us":70132,"drs_written":2,"lbm_read_time_us":3722354,"lbm_reads_1-10_ms":131,"lbm_reads_10-100_ms":78,"lbm_reads_gt_100_ms":5,"lbm_reads_lt_1ms":6604,"lbm_write_time_us":576685,"lbm_writes_1-10_ms":78,"lbm_writes_10-100_ms":7,"lbm_writes_lt_1ms":5091,"num_input_rowsets":3,"rows_written":2917756,"spinlock_wait_cycles":46715008,"thread_start_us":691,"threads_started":10} I0107 12:46:10.865384 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 62636254 bytes on disk I0107 12:46:11.009548 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 8.656s user 2.362s sys 0.157s Metrics: {"bytes_written":27466160,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":79251,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":300820,"lbm_writes_1-10_ms":45,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":2716,"reinsert_count":0,"rows_written":1556938,"thread_start_us":48,"threads_started":1,"update_count":264} I0107 12:46:11.009686 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 58734893 bytes of WAL I0107 12:46:11.009747 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 7 log segments from log reader I0107 12:46:11.009759 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000053 (ops 307380-313275) I0107 12:46:11.009804 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000054 (ops 313276-319139) I0107 12:46:11.009824 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000055 (ops 319140-325061) I0107 12:46:11.009861 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000056 (ops 325062-330992) I0107 12:46:11.009878 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000057 (ops 330993-336860) I0107 12:46:11.009888 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000058 (ops 336861-342735) I0107 12:46:11.009898 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000059 (ops 342736-348626) I0107 12:46:11.022377 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.013s user 0.000s sys 0.011s Metrics: {} I0107 12:46:11.022517 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 30860393 bytes on disk I0107 12:46:11.022616 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0107 12:46:11.022677 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.625464 I0107 12:46:11.200755 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.335s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":335161,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:46:11.200891 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.048822 I0107 12:46:11.615558 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 14.853s user 3.609s sys 0.410s Metrics: {"bytes_written":51417141,"cfile_cache_hit":97,"cfile_cache_hit_bytes":11877606,"cfile_cache_miss":6805,"cfile_cache_miss_bytes":199991834,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":1432,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":83668,"drs_written":2,"lbm_read_time_us":3247390,"lbm_reads_1-10_ms":84,"lbm_reads_10-100_ms":84,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":6686,"lbm_write_time_us":577129,"lbm_writes_1-10_ms":116,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":5059,"num_input_rowsets":3,"rows_written":2921980,"spinlock_wait_cycles":4700544,"thread_start_us":562,"threads_started":10} I0107 12:46:11.615873 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.039221 I0107 12:46:12.330621 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 1.130s user 0.613s sys 0.031s Metrics: {"bytes_written":9632606,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":142777,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":326148,"lbm_writes_1-10_ms":13,"lbm_writes_10-100_ms":11,"lbm_writes_lt_1ms":961,"reinsert_count":0,"rows_written":547282,"thread_start_us":31,"threads_started":1,"update_count":1388} I0107 12:46:12.330744 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 42007285 bytes of WAL I0107 12:46:12.330801 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 5 log segments from log reader I0107 12:46:12.330809 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000024 (ops 285459-297840) I0107 12:46:12.330834 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000025 (ops 297841-310282) I0107 12:46:12.330850 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000026 (ops 310283-322721) I0107 12:46:12.330862 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000027 (ops 322722-335102) I0107 12:46:12.330871 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000028 (ops 335103-347501) I0107 12:46:12.341315 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.010s user 0.000s sys 0.009s Metrics: {} I0107 12:46:12.341416 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.004288 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 360000 Mean: 2098.74 Percentiles: 0% (min) = 778 25% = 1439 50% (med) = 1625 75% = 1883 95% = 2652 99% = 9096 99.9% = 53248 99.99% = 297728 100% (max) = 3087444 *NOTE: some values were greater than highest trackable value I0107 12:46:18.653982 24384 linked_list-test-util.h:519] Written 18000000 rows in chain I0107 12:46:22.616921 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 10.990s user 2.639s sys 0.241s Metrics: {"bytes_written":31160465,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":10819,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":26336,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1711609,"lbm_writes_1-10_ms":96,"lbm_writes_10-100_ms":27,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":3008,"reinsert_count":0,"rows_written":1768259,"thread_start_us":29,"threads_started":1,"update_count":1139} I0107 12:46:22.617178 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 75697845 bytes of WAL I0107 12:46:22.617270 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 9 log segments from log reader I0107 12:46:22.617322 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000052 (ops 301737-307665) I0107 12:46:22.617410 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000053 (ops 307666-313549) I0107 12:46:22.617491 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000054 (ops 313550-319444) I0107 12:46:22.618070 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000055 (ops 319445-325357) I0107 12:46:22.618362 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000056 (ops 325358-331188) I0107 12:46:22.618468 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000057 (ops 331189-337049) I0107 12:46:22.618652 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000058 (ops 337050-343060) I0107 12:46:22.618714 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000059 (ops 343061-348965) I0107 12:46:22.618782 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000060 (ops 348966-354877) I0107 12:46:22.723855 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.107s user 0.001s sys 0.023s Metrics: {} I0107 12:46:22.725746 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.619824 I0107 12:46:29.194384 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 16.853s user 3.914s sys 0.424s Metrics: {"bytes_written":52379809,"cfile_cache_hit":124,"cfile_cache_hit_bytes":15942136,"cfile_cache_miss":6896,"cfile_cache_miss_bytes":199883572,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":14879,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":43485,"drs_written":2,"lbm_read_time_us":3247004,"lbm_reads_1-10_ms":116,"lbm_reads_10-100_ms":68,"lbm_reads_gt_100_ms":7,"lbm_reads_lt_1ms":6757,"lbm_write_time_us":738127,"lbm_writes_1-10_ms":122,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":5148,"num_input_rowsets":3,"rows_written":2977368,"spinlock_wait_cycles":46278400,"thread_start_us":417,"threads_started":10} I0107 12:46:29.194582 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.040713 I0107 12:46:40.157302 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 10.959s user 2.810s sys 0.181s Metrics: {"bytes_written":33450477,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":77475,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2432707,"lbm_writes_1-10_ms":69,"lbm_writes_10-100_ms":15,"lbm_writes_gt_100_ms":5,"lbm_writes_lt_1ms":3271,"reinsert_count":0,"rows_written":1896250,"thread_start_us":44,"threads_started":1,"update_count":7284} I0107 12:46:40.157532 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.622742 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 370000 Mean: 2105.31 Percentiles: 0% (min) = 778 25% = 1442 50% (med) = 1629 75% = 1888 95% = 2660 99% = 9176 99.9% = 53984 99.99% = 297728 100% (max) = 3087444 *NOTE: some values were greater than highest trackable value I0107 12:46:43.426730 24384 linked_list-test-util.h:519] Written 18500000 rows in chain I0107 12:46:58.807793 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 47.785s user 10.338s sys 1.146s Metrics: {"bytes_written":140033801,"cfile_cache_hit":255,"cfile_cache_hit_bytes":33802166,"cfile_cache_miss":18505,"cfile_cache_miss_bytes":543501144,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":11625,"dirs.run_cpu_time_us":525,"dirs.run_wall_time_us":349083,"drs_written":5,"lbm_read_time_us":10803217,"lbm_reads_1-10_ms":341,"lbm_reads_10-100_ms":210,"lbm_reads_gt_100_ms":17,"lbm_reads_lt_1ms":18021,"lbm_write_time_us":1667266,"lbm_writes_1-10_ms":261,"lbm_writes_10-100_ms":12,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":13835,"num_input_rowsets":5,"rows_written":7958124,"spinlock_wait_cycles":259377664,"thread_start_us":692,"threads_started":5} I0107 12:46:58.807981 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 170270622 bytes on disk I0107 12:46:58.808161 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":31,"lbm_reads_lt_1ms":20} I0107 12:46:58.808243 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.511117 I0107 12:47:00.531549 29939 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1115991, attempt_no=0}) took 1713ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":12,"cfile_cache_hit":12,"cfile_cache_hit_bytes":84626,"cfile_cache_miss":30,"cfile_cache_miss_bytes":132753,"cfile_init":2,"lbm_read_time_us":1712235,"lbm_reads_1-10_ms":6,"lbm_reads_10-100_ms":20,"lbm_reads_gt_100_ms":5,"lbm_reads_lt_1ms":7,"num_ops":28,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":81,"raft.queue_time_us":45,"raft.run_cpu_time_us":27,"raft.run_wall_time_us":28,"replication_time_us":394,"thread_start_us":31,"threads_started":1,"wal-append.queue_time_us":71}]]} I0107 12:47:02.219961 29939 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1116002, attempt_no=0}) took 1125ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":9,"cfile_cache_hit":31,"cfile_cache_hit_bytes":197915,"cfile_cache_miss":18,"cfile_cache_miss_bytes":73908,"lbm_read_time_us":1123874,"lbm_reads_10-100_ms":8,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":6,"num_ops":34,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":217,"raft.queue_time_us":27,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":68,"replication_time_us":440,"spinlock_wait_cycles":65664,"thread_start_us":18,"threads_started":1}]]} I0107 12:47:03.010291 29888 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.3:43359 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1115996, attempt_no=0}) took 3938ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":10,"cfile_cache_hit":1935,"cfile_cache_hit_bytes":8013431,"cfile_cache_miss":3554,"cfile_cache_miss_bytes":14873771,"lbm_read_time_us":3856852,"lbm_reads_1-10_ms":42,"lbm_reads_10-100_ms":63,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":3441,"num_ops":10130,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":10188,"prepare.run_wall_time_us":24126,"raft.queue_time_us":153,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":145,"replication_time_us":2114,"spinlock_wait_cycles":384}]]} I0107 12:47:09.308648 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 46.582s user 10.102s sys 1.045s Metrics: {"bytes_written":141514879,"cfile_cache_hit":266,"cfile_cache_hit_bytes":35317237,"cfile_cache_miss":18690,"cfile_cache_miss_bytes":547770207,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":957,"dirs.run_cpu_time_us":599,"dirs.run_wall_time_us":448296,"drs_written":5,"lbm_read_time_us":12607195,"lbm_reads_1-10_ms":316,"lbm_reads_10-100_ms":229,"lbm_reads_gt_100_ms":23,"lbm_reads_lt_1ms":18210,"lbm_write_time_us":1505837,"lbm_writes_1-10_ms":260,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":13989,"num_input_rowsets":5,"rows_written":8043760,"spinlock_wait_cycles":132457216,"thread_start_us":699,"threads_started":13} I0107 12:47:09.308851 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.513538 I0107 12:47:10.858477 30074 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1123370, attempt_no=0}) took 1497ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1496796,"cfile_cache_hit":7,"cfile_cache_hit_bytes":43970,"num_ops":5,"prepare.queue_time_us":7,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":150,"raft.run_cpu_time_us":35,"raft.run_wall_time_us":35,"replication_time_us":604}]]} I0107 12:47:13.636420 30074 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1123520, attempt_no=0}) took 1877ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1874780,"cfile_cache_hit":5,"cfile_cache_hit_bytes":32399,"num_ops":4,"prepare.queue_time_us":34,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":87,"raft.queue_time_us":49,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":576,"replication_time_us":2114,"spinlock_wait_cycles":4480,"thread_start_us":75,"threads_started":3}]]} I0107 12:47:13.638357 30074 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.1:34177 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1123525, attempt_no=0}) took 1778ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1775134,"cfile_cache_hit":379,"cfile_cache_hit_bytes":1559533,"num_ops":430,"prepare.queue_time_us":33,"prepare.run_cpu_time_us":464,"prepare.run_wall_time_us":464,"raft.queue_time_us":42,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":47,"replication_time_us":660,"spinlock_wait_cycles":14720}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) I0107 12:47:16.377112 24384 linked_list-test-util.h:519] Written 19000000 rows in chain Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 380000 Mean: 2133.41 Percentiles: 0% (min) = 778 25% = 1444 50% (med) = 1633 75% = 1894 95% = 2676 99% = 9248 99.9% = 54848 99.99% = 316672 100% (max) = 3087444 *NOTE: some values were greater than highest trackable value I0107 12:47:30.540906 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 50.383s user 10.563s sys 1.078s Metrics: {"bytes_written":145663350,"cfile_cache_hit":285,"cfile_cache_hit_bytes":38206738,"cfile_cache_miss":19231,"cfile_cache_miss_bytes":562014212,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":25549,"dirs.run_cpu_time_us":533,"dirs.run_wall_time_us":247479,"drs_written":5,"lbm_read_time_us":12607904,"lbm_reads_1-10_ms":325,"lbm_reads_10-100_ms":267,"lbm_reads_gt_100_ms":15,"lbm_reads_lt_1ms":18712,"lbm_write_time_us":1451116,"lbm_writes_1-10_ms":259,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":14410,"num_input_rowsets":5,"rows_written":8281253,"spinlock_wait_cycles":422451200,"thread_start_us":182,"threads_started":5} I0107 12:47:30.541087 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 75332692 bytes of WAL I0107 12:47:30.541296 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 9 log segments from log reader I0107 12:47:30.541308 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000053 (ops 307856-313709) I0107 12:47:30.541342 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000054 (ops 313710-319609) I0107 12:47:30.541366 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000055 (ops 319610-325496) I0107 12:47:30.541388 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000056 (ops 325497-331430) I0107 12:47:30.541411 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000057 (ops 331431-337356) I0107 12:47:30.541431 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000058 (ops 337357-343190) I0107 12:47:30.541452 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000059 (ops 343191-349106) I0107 12:47:30.541473 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000060 (ops 349107-354961) I0107 12:47:30.541534 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000061 (ops 354962-360854) I0107 12:47:30.608183 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.067s user 0.000s sys 0.023s Metrics: {} I0107 12:47:30.608340 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.446014 I0107 12:47:33.837117 30233 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1144579, attempt_no=0}) took 2695ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":2692385,"cfile_cache_hit":7,"cfile_cache_hit_bytes":44921,"num_ops":5,"prepare.queue_time_us":14,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":26,"raft.run_cpu_time_us":16,"raft.run_wall_time_us":16,"replication_time_us":3221,"thread_start_us":36,"threads_started":1,"wal-append.queue_time_us":61}]]} I0107 12:47:33.878057 30182 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.208.2:33061 (ReqId={client: 38ca74434bcb40b7a65933bd12dc4ffd, seq_no=1144583, attempt_no=0}) took 1841ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1838008,"cfile_cache_hit":356,"cfile_cache_hit_bytes":1477915,"num_ops":406,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":570,"prepare.run_wall_time_us":570,"raft.queue_time_us":74,"raft.run_cpu_time_us":35,"raft.run_wall_time_us":34,"replication_time_us":724,"spinlock_wait_cycles":1280,"thread_start_us":49,"threads_started":2}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 390000 Mean: 2142.9 Percentiles: 0% (min) = 778 25% = 1446 50% (med) = 1636 75% = 1898 95% = 2692 99% = 9280 99.9% = 54848 99.99% = 335872 100% (max) = 3087444 *NOTE: some values were greater than highest trackable value I0107 12:47:42.733299 24384 linked_list-test-util.h:519] Written 19500000 rows in chain I0107 12:47:44.979089 24384 linked_list-test-util.h:528] Sample at HT timestamp: 6465249546147262464 Inserted count: 19547200 I0107 12:47:44.979239 24384 linked_list-test-util.h:532] Finished inserting list. Added 19547200 in chain I0107 12:47:44.979282 24384 linked_list-test-util.h:533] Last entries inserted had keys: I0107 12:47:44.979317 24384 linked_list-test-util.h:535] 0: -3791051758930886656 I0107 12:47:44.979357 24384 linked_list-test-util.h:535] 1: 4079224589352435713 I0107 12:47:44.979391 24384 linked_list-test-util.h:535] 2: -6496961661097082878 I0107 12:47:44.979434 24384 linked_list-test-util.h:535] 3: 1373736899651239939 I0107 12:47:44.979480 24384 linked_list-test-util.h:535] 4: 5453102226492096516 I0107 12:47:44.979538 24384 linked_list-test-util.h:535] 5: -5123365498934132731 I0107 12:47:44.979575 24384 linked_list-test-util.h:535] 6: 2747333061814190086 I0107 12:47:44.979609 24384 linked_list-test-util.h:535] 7: -7828853188635328505 I0107 12:47:44.979645 24384 linked_list-test-util.h:535] 8: -3749769336771182584 I0107 12:47:44.979684 24384 linked_list-test-util.h:535] 9: 4120929223977140233 I0107 12:47:44.979730 24384 linked_list-test-util.h:535] 10: -6455257026472378358 I0107 12:47:44.979770 24384 linked_list-test-util.h:535] 11: 1415300796787654667 I0107 12:47:44.979807 24384 linked_list-test-util.h:535] 12: 5494525386140090380 I0107 12:47:44.979846 24384 linked_list-test-util.h:535] 13: -5081660864309428211 I0107 12:47:44.979882 24384 linked_list-test-util.h:535] 14: 2788896958950604814 I0107 12:47:44.979919 24384 linked_list-test-util.h:535] 15: -7787148554010623985 I0107 12:47:44.979960 24384 linked_list-test-util.h:535] 16: -3708064702146478064 I0107 12:47:44.979997 24384 linked_list-test-util.h:535] 17: 4162493121113554961 I0107 12:47:44.980033 24384 linked_list-test-util.h:535] 18: -6413552391847673838 I0107 12:47:44.980068 24384 linked_list-test-util.h:535] 19: 1456723956435648531 I0107 12:47:44.980104 24384 linked_list-test-util.h:535] 20: 5536089283276505108 I0107 12:47:44.980139 24384 linked_list-test-util.h:535] 21: -5039956229684723691 I0107 12:47:44.980175 24384 linked_list-test-util.h:535] 22: 2830601593575309334 I0107 12:47:44.980212 24384 linked_list-test-util.h:535] 23: -7745866131850919913 I0107 12:47:44.980247 24384 linked_list-test-util.h:535] 24: -3666360067521773544 I0107 12:47:44.980280 24384 linked_list-test-util.h:535] 25: 4204197755738259481 I0107 12:47:44.980315 24384 linked_list-test-util.h:535] 26: -6371988494711259110 I0107 12:47:44.980347 24384 linked_list-test-util.h:535] 27: -2292763905358823397 I0107 12:47:44.980381 24384 linked_list-test-util.h:535] 28: 5577793917901209628 I0107 12:47:44.980417 24384 linked_list-test-util.h:535] 29: -4998392332548308963 I0107 12:47:44.980458 24384 linked_list-test-util.h:535] 30: 2872306228200013854 I0107 12:47:44.980510 24384 linked_list-test-util.h:535] 31: 6951390080064159775 I0107 12:47:44.980548 24384 linked_list-test-util.h:535] 32: -3624796170385358816 I0107 12:47:44.980582 24384 linked_list-test-util.h:535] 33: 4245902390362964001 I0107 12:47:44.980615 24384 linked_list-test-util.h:535] 34: -6330283860086554590 I0107 12:47:44.980651 24384 linked_list-test-util.h:535] 35: -2251200008222408669 I0107 12:47:44.980684 24384 linked_list-test-util.h:535] 36: 5619498552525914148 I0107 12:47:44.980717 24384 linked_list-test-util.h:535] 37: -4956687697923604443 I0107 12:47:44.980751 24384 linked_list-test-util.h:535] 38: 2913588650359717926 I0107 12:47:44.980782 24384 linked_list-test-util.h:535] 39: 6993094714688864295 I0107 12:47:44.980818 24384 linked_list-test-util.h:535] 40: -3583091535760654296 I0107 12:47:44.980852 24384 linked_list-test-util.h:535] 41: 4287466287499378729 I0107 12:47:44.980885 24384 linked_list-test-util.h:535] 42: -6288860700438560726 I0107 12:47:44.980922 24384 linked_list-test-util.h:535] 43: -2209495373597704149 I0107 12:47:44.980960 24384 linked_list-test-util.h:535] 44: 5661062449662328876 I0107 12:47:44.980995 24384 linked_list-test-util.h:535] 45: -4914983063298899923 I0107 12:47:44.981029 24384 linked_list-test-util.h:535] 46: 2955293284984422446 I0107 12:47:44.981062 24384 linked_list-test-util.h:535] 47: 7034658611825279023 I0107 12:47:44.981094 24384 linked_list-test-util.h:535] 48: -3541386901135949776 I0107 12:47:44.981127 24384 linked_list-test-util.h:535] 49: 4329170922124083249 I0107 12:47:45.225585 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578429464976514 usec, L: 1 Expected Rows: 7530250 I0107 12:47:45.511669 24384 linked_list-test-util.h:639] Calling callback on tserver 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:47:46.889780 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 48.081s user 9.542s sys 0.996s Metrics: {"bytes_written":135380590,"cfile_cache_hit":193,"cfile_cache_hit_bytes":24959481,"cfile_cache_miss":17937,"cfile_cache_miss_bytes":536977123,"cfile_init":16,"delta_iterators_relevant":8,"dirs.queue_time_us":4021,"dirs.run_cpu_time_us":551,"dirs.run_wall_time_us":320706,"drs_written":4,"lbm_read_time_us":16276795,"lbm_reads_1-10_ms":276,"lbm_reads_10-100_ms":301,"lbm_reads_gt_100_ms":27,"lbm_reads_lt_1ms":17397,"lbm_write_time_us":2776071,"lbm_writes_1-10_ms":257,"lbm_writes_10-100_ms":35,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":13333,"num_input_rowsets":4,"rows_written":7695443,"spinlock_wait_cycles":146440448,"thread_start_us":194,"threads_started":5,"wal-append.queue_time_us":70} I0107 12:47:46.889920 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 165609443 bytes on disk I0107 12:47:46.890038 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":20,"lbm_reads_lt_1ms":16} I0107 12:47:46.890110 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.040566 I0107 12:47:47.981317 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 1.091s user 0.495s sys 0.020s Metrics: {"bytes_written":7661016,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":57,"dirs.run_wall_time_us":57,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":347231,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":765,"rows_written":432259,"thread_start_us":64,"threads_started":2,"wal-append.queue_time_us":69} I0107 12:47:47.981446 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 33553966 bytes of WAL I0107 12:47:47.981520 24731 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 4 log segments from log reader I0107 12:47:47.981530 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000027 (ops 323164-335646) I0107 12:47:47.981557 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000028 (ops 335647-347980) I0107 12:47:47.981658 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000029 (ops 347981-360440) I0107 12:47:47.981722 24731 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000030 (ops 360441-372768) I0107 12:47:47.990912 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.009s user 0.000s sys 0.009s Metrics: {} I0107 12:47:47.991047 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 8609324 bytes on disk I0107 12:47:47.991168 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0107 12:47:47.991250 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.001635 I0107 12:47:53.123517 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 43.811s user 9.128s sys 0.889s Metrics: {"bytes_written":136035578,"cfile_cache_hit":168,"cfile_cache_hit_bytes":21234984,"cfile_cache_miss":18056,"cfile_cache_miss_bytes":543299948,"cfile_init":18,"delta_iterators_relevant":8,"dirs.queue_time_us":11704,"dirs.run_cpu_time_us":435,"dirs.run_wall_time_us":193165,"drs_written":4,"lbm_read_time_us":16219004,"lbm_reads_1-10_ms":284,"lbm_reads_10-100_ms":301,"lbm_reads_gt_100_ms":26,"lbm_reads_lt_1ms":17517,"lbm_write_time_us":2797141,"lbm_writes_1-10_ms":223,"lbm_writes_10-100_ms":44,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":13427,"num_input_rowsets":4,"rows_written":7733404,"spinlock_wait_cycles":64884352,"thread_start_us":531,"threads_started":13,"wal-append.queue_time_us":52} I0107 12:47:53.123678 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 256499050 bytes on disk I0107 12:47:53.509210 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.385s user 0.000s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":385142,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":10,"lbm_reads_lt_1ms":17} I0107 12:47:53.509326 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.037844 I0107 12:47:54.004838 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.495s user 0.434s sys 0.027s Metrics: {"bytes_written":6908504,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":73797,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14543,"lbm_writes_lt_1ms":703,"rows_written":389289,"thread_start_us":72,"threads_started":2,"wal-append.queue_time_us":57} I0107 12:47:54.004953 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 25174932 bytes of WAL I0107 12:47:54.005021 24482 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 3 log segments from log reader I0107 12:47:54.005036 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000028 (ops 334945-347341) I0107 12:47:54.005066 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000029 (ops 347342-359773) I0107 12:47:54.005087 24482 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000030 (ops 359774-372127) I0107 12:47:54.011543 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.007s user 0.001s sys 0.006s Metrics: {} I0107 12:47:54.011663 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 62781870 bytes on disk I0107 12:47:54.173941 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.162s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":162083,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":4} I0107 12:47:54.174119 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.002072 I0107 12:48:03.495343 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 32.887s user 7.311s sys 0.656s Metrics: {"bytes_written":111641551,"cfile_cache_hit":150,"cfile_cache_hit_bytes":18756939,"cfile_cache_miss":14812,"cfile_cache_miss_bytes":443998757,"cfile_init":18,"delta_iterators_relevant":8,"dirs.queue_time_us":1355,"dirs.run_cpu_time_us":563,"dirs.run_wall_time_us":503981,"drs_written":4,"lbm_read_time_us":16283958,"lbm_reads_1-10_ms":192,"lbm_reads_10-100_ms":344,"lbm_reads_gt_100_ms":25,"lbm_reads_lt_1ms":14323,"lbm_write_time_us":1227102,"lbm_writes_1-10_ms":66,"lbm_writes_10-100_ms":11,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":11175,"num_input_rowsets":4,"rows_written":6347934,"spinlock_wait_cycles":98386688,"thread_start_us":703,"threads_started":13,"wal-append.queue_time_us":59} I0107 12:48:03.495579 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(77e67457a0334f92a22f52fe739203dd): perf score=0.035767 I0107 12:48:04.285226 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.790s user 0.362s sys 0.014s Metrics: {"bytes_written":6036011,"cfile_init":1,"dirs.queue_time_us":16,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":69730,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":363337,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":613,"rows_written":339671,"thread_start_us":33,"threads_started":1,"wal-append.queue_time_us":73} I0107 12:48:04.285326 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(77e67457a0334f92a22f52fe739203dd): free 25209813 bytes of WAL I0107 12:48:04.285368 24607 log_reader.cc:379] T 77e67457a0334f92a22f52fe739203dd: removed 3 log segments from log reader I0107 12:48:04.285382 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000028 (ops 334984-347384) I0107 12:48:04.285403 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000029 (ops 347385-359823) I0107 12:48:04.285413 24607 log.cc:1062] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/77e67457a0334f92a22f52fe739203dd/wal-000000030 (ops 359824-372249) I0107 12:48:04.291923 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.007s user 0.000s sys 0.007s Metrics: {} I0107 12:48:04.292035 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 261816490 bytes on disk I0107 12:48:04.958166 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.666s user 0.001s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":665737,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":11,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":13} I0107 12:48:04.958312 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd): perf score=1.002573 I0107 12:48:09.604792 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 15.431s user 3.636s sys 0.281s Metrics: {"bytes_written":57259996,"cfile_cache_hit":77,"cfile_cache_hit_bytes":8960826,"cfile_cache_miss":7601,"cfile_cache_miss_bytes":228621566,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":501,"dirs.run_cpu_time_us":330,"dirs.run_wall_time_us":112447,"drs_written":2,"lbm_read_time_us":9908612,"lbm_reads_1-10_ms":80,"lbm_reads_10-100_ms":215,"lbm_reads_gt_100_ms":15,"lbm_reads_lt_1ms":7339,"lbm_write_time_us":172198,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":5771,"num_input_rowsets":3,"rows_written":3257427,"thread_start_us":744,"threads_started":11,"wal-append.queue_time_us":67} I0107 12:48:09.605015 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.036904 I0107 12:48:09.998961 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.394s user 0.308s sys 0.021s Metrics: {"bytes_written":5970004,"cfile_init":1,"dirs.queue_time_us":15,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":80404,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":34997,"lbm_writes_1-10_ms":4,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":606,"rows_written":335925,"spinlock_wait_cycles":4992,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":134} I0107 12:48:09.999087 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 25186083 bytes of WAL I0107 12:48:09.999131 24482 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 3 log segments from log reader I0107 12:48:09.999141 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000028 (ops 335834-348277) I0107 12:48:09.999187 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000029 (ops 348278-360708) I0107 12:48:09.999228 24482 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000030 (ops 360709-373165) I0107 12:48:10.008343 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.009s user 0.000s sys 0.008s Metrics: {} I0107 12:48:10.008512 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 70340981 bytes on disk I0107 12:48:10.216889 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.208s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":208176,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:48:10.217020 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 6660645 bytes on disk I0107 12:48:10.217124 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0107 12:48:10.217175 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.002611 I0107 12:48:15.803031 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 27.812s user 3.868s sys 0.319s Metrics: {"bytes_written":57259683,"cfile_cache_hit":93,"cfile_cache_hit_bytes":11333115,"cfile_cache_miss":7591,"cfile_cache_miss_bytes":226055617,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":721,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":92907,"drs_written":2,"lbm_read_time_us":19863744,"lbm_reads_1-10_ms":167,"lbm_reads_10-100_ms":398,"lbm_reads_gt_100_ms":27,"lbm_reads_lt_1ms":7043,"lbm_write_time_us":2681481,"lbm_writes_1-10_ms":33,"lbm_writes_10-100_ms":54,"lbm_writes_gt_100_ms":7,"lbm_writes_lt_1ms":5681,"num_input_rowsets":3,"rows_written":3257427,"spinlock_wait_cycles":4224,"thread_start_us":668,"threads_started":11,"wal-append.queue_time_us":84} I0107 12:48:15.803238 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.043250 I0107 12:48:16.633422 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.830s user 0.410s sys 0.021s Metrics: {"bytes_written":6824886,"cfile_init":1,"dirs.queue_time_us":76,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":329168,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":684,"rows_written":384597,"thread_start_us":63,"threads_started":2,"wal-append.queue_time_us":60} I0107 12:48:16.633544 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 25179328 bytes of WAL I0107 12:48:16.633615 24731 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 3 log segments from log reader I0107 12:48:16.633628 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000028 (ops 335925-348385) I0107 12:48:16.633661 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000029 (ops 348386-360835) I0107 12:48:16.633682 24731 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000030 (ops 360836-373320) I0107 12:48:16.639762 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.006s user 0.000s sys 0.006s Metrics: {} I0107 12:48:16.639866 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.002120 I0107 12:48:20.284497 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 15.326s user 3.452s sys 0.252s Metrics: {"bytes_written":57260301,"cfile_cache_hit":79,"cfile_cache_hit_bytes":9251387,"cfile_cache_miss":7605,"cfile_cache_miss_bytes":228542323,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":426,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":99446,"drs_written":2,"lbm_read_time_us":10310247,"lbm_reads_1-10_ms":79,"lbm_reads_10-100_ms":206,"lbm_reads_gt_100_ms":16,"lbm_reads_lt_1ms":7352,"lbm_write_time_us":158618,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":5774,"num_input_rowsets":3,"rows_written":3257427,"thread_start_us":658,"threads_started":11,"wal-append.queue_time_us":83} I0107 12:48:20.284641 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 70340395 bytes on disk I0107 12:48:20.496954 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.212s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":212117,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:48:20.497063 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87): perf score=0.035602 I0107 12:48:20.887240 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.390s user 0.327s sys 0.025s Metrics: {"bytes_written":4996270,"cfile_init":1,"dirs.queue_time_us":26,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":27732,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11887,"lbm_writes_lt_1ms":513,"rows_written":280537,"thread_start_us":35,"threads_started":1,"wal-append.queue_time_us":64} I0107 12:48:20.887399 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(30019a15e66f4115b11a50dbcaf93a87): free 16791253 bytes of WAL I0107 12:48:20.887451 24607 log_reader.cc:379] T 30019a15e66f4115b11a50dbcaf93a87: removed 2 log segments from log reader I0107 12:48:20.887461 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000029 (ops 347502-359925) I0107 12:48:20.887490 24607 log.cc:1062] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/30019a15e66f4115b11a50dbcaf93a87/wal-000000030 (ops 359926-372309) I0107 12:48:20.893263 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.006s user 0.000s sys 0.005s Metrics: {} I0107 12:48:20.893435 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 69576445 bytes on disk I0107 12:48:20.987500 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.094s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":93826,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":8} I0107 12:48:20.987624 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87): perf score=1.003174 I0107 12:48:25.827849 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 15.611s user 3.486s sys 0.307s Metrics: {"bytes_written":57264600,"cfile_cache_hit":77,"cfile_cache_hit_bytes":8951182,"cfile_cache_miss":7603,"cfile_cache_miss_bytes":228989046,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":530,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":111654,"drs_written":2,"lbm_read_time_us":10676617,"lbm_reads_1-10_ms":70,"lbm_reads_10-100_ms":214,"lbm_reads_gt_100_ms":15,"lbm_reads_lt_1ms":7348,"lbm_write_time_us":153605,"lbm_writes_1-10_ms":3,"lbm_writes_lt_1ms":5772,"num_input_rowsets":3,"rows_written":3257905,"thread_start_us":634,"threads_started":11,"wal-append.queue_time_us":64} I0107 12:48:25.828069 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 70413623 bytes on disk I0107 12:48:26.024046 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.196s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":195774,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} I0107 12:48:26.024171 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.034283 I0107 12:48:26.726838 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 10.087s user 3.431s sys 0.188s Metrics: {"bytes_written":57264719,"cfile_cache_hit":6587,"cfile_cache_hit_bytes":187286992,"cfile_cache_miss":1093,"cfile_cache_miss_bytes":50439754,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":197,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":160876,"drs_written":2,"lbm_read_time_us":925526,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":16,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":1112,"lbm_write_time_us":4362005,"lbm_writes_1-10_ms":65,"lbm_writes_10-100_ms":97,"lbm_writes_gt_100_ms":7,"lbm_writes_lt_1ms":5606,"num_input_rowsets":3,"rows_written":3257905,"spinlock_wait_cycles":1664,"thread_start_us":514,"threads_started":11,"wal-append.queue_time_us":50} I0107 12:48:26.726966 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 70412941 bytes on disk I0107 12:48:26.727105 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8} I0107 12:48:26.727156 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 28590883 bytes on disk I0107 12:48:26.771342 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.044s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44048,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":1} I0107 12:48:26.771471 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.037712 I0107 12:48:27.991065 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 1.967s user 1.410s sys 0.090s Metrics: {"bytes_written":19954956,"cfile_init":1,"dirs.queue_time_us":70,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":36909,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":47769,"lbm_writes_lt_1ms":1993,"rows_written":1121456,"thread_start_us":63,"threads_started":2,"wal-append.queue_time_us":55} I0107 12:48:27.991291 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 50299454 bytes of WAL I0107 12:48:27.991369 24482 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 6 log segments from log reader I0107 12:48:27.991412 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000061 (ops 354878-360752) I0107 12:48:27.991581 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000062 (ops 360753-366686) I0107 12:48:27.991653 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000063 (ops 366687-372619) I0107 12:48:27.991713 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000064 (ops 372620-378510) I0107 12:48:27.991739 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000065 (ops 378511-384451) I0107 12:48:27.991760 24482 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000066 (ops 384452-390235) I0107 12:48:28.007086 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.016s user 0.001s sys 0.014s Metrics: {} I0107 12:48:28.007259 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 22167655 bytes on disk I0107 12:48:28.007423 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0107 12:48:28.007529 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.678158 I0107 12:48:30.086611 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 3.315s user 1.476s sys 0.085s Metrics: {"bytes_written":22134157,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":114823,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1156514,"lbm_writes_1-10_ms":16,"lbm_writes_10-100_ms":13,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":2178,"rows_written":1244996,"thread_start_us":64,"threads_started":2,"wal-append.queue_time_us":70} I0107 12:48:30.086771 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 58813357 bytes of WAL I0107 12:48:30.086838 24731 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 7 log segments from log reader I0107 12:48:30.086853 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000060 (ops 348627-354525) I0107 12:48:30.086905 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000061 (ops 354526-360418) I0107 12:48:30.086971 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000062 (ops 360419-366263) I0107 12:48:30.087003 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000063 (ops 366264-372185) I0107 12:48:30.087061 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000064 (ops 372186-378087) I0107 12:48:30.087096 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000065 (ops 378088-384007) I0107 12:48:30.087127 24731 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000066 (ops 384008-389980) I0107 12:48:30.100353 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.014s user 0.000s sys 0.013s Metrics: {} I0107 12:48:30.100491 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 24702753 bytes on disk I0107 12:48:30.100627 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0107 12:48:30.100724 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.653159 I0107 12:48:36.039594 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 15.052s user 3.504s sys 0.322s Metrics: {"bytes_written":57264197,"cfile_cache_hit":65,"cfile_cache_hit_bytes":7171295,"cfile_cache_miss":7615,"cfile_cache_miss_bytes":231014577,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":182945,"drs_written":2,"lbm_read_time_us":9251853,"lbm_reads_1-10_ms":103,"lbm_reads_10-100_ms":164,"lbm_reads_gt_100_ms":18,"lbm_reads_lt_1ms":7374,"lbm_write_time_us":179039,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":5771,"num_input_rowsets":3,"rows_written":3257905,"thread_start_us":526,"threads_started":11,"wal-append.queue_time_us":74} I0107 12:48:36.039795 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 70413540 bytes on disk I0107 12:48:36.039935 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8} I0107 12:48:37.833609 24384 linked_list-test-util.h:876] Done collecting results (7530250 rows in 52321.9ms) I0107 12:48:38.861021 24384 linked_list-test-util.h:667] Snapshot: P: 1578429464976514 usec, L: 1 verified. Result: OK I0107 12:48:38.861575 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578429764977164 usec, L: 1 Expected Rows: 14012700 I0107 12:48:39.634757 24384 linked_list-test-util.h:639] Calling callback on tserver c0dcf07e91a6459fb83f90f61d363e44 I0107 12:48:40.293365 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling FlushMRSOp(16e58229a0c648eda40d4376454f2981): perf score=0.033371 I0107 12:48:41.896951 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: FlushMRSOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 1.603s user 1.017s sys 0.056s Metrics: {"bytes_written":15767879,"cfile_init":1,"dirs.queue_time_us":75,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":72961,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":258765,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":7,"lbm_writes_lt_1ms":1560,"rows_written":883929,"thread_start_us":76,"threads_started":2,"wal-append.queue_time_us":67} I0107 12:48:41.897161 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling LogGCOp(16e58229a0c648eda40d4376454f2981): free 42249049 bytes of WAL I0107 12:48:41.897250 24607 log_reader.cc:379] T 16e58229a0c648eda40d4376454f2981: removed 5 log segments from log reader I0107 12:48:41.897292 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000062 (ops 360855-366811) I0107 12:48:41.897372 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000063 (ops 366812-372717) I0107 12:48:41.897469 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000064 (ops 372718-378669) I0107 12:48:41.897527 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000065 (ops 378670-384620) I0107 12:48:41.897586 24607 log.cc:1062] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal/wals/16e58229a0c648eda40d4376454f2981/wal-000000066 (ops 384621-390605) I0107 12:48:41.910905 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: LogGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.014s user 0.001s sys 0.011s Metrics: {} I0107 12:48:41.911006 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.682082 I0107 12:49:06.173947 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 38.166s user 10.035s sys 0.807s Metrics: {"bytes_written":161179337,"cfile_cache_hit":52,"cfile_cache_hit_bytes":2134745,"cfile_cache_miss":21552,"cfile_cache_miss_bytes":667382695,"cfile_init":23,"delta_iterators_relevant":12,"dirs.queue_time_us":7885,"dirs.run_cpu_time_us":505,"dirs.run_wall_time_us":419779,"drs_written":5,"lbm_read_time_us":23598117,"lbm_reads_1-10_ms":272,"lbm_reads_10-100_ms":486,"lbm_reads_gt_100_ms":33,"lbm_reads_lt_1ms":20853,"lbm_write_time_us":555562,"lbm_writes_1-10_ms":4,"lbm_writes_10-100_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":16230,"num_input_rowsets":6,"rows_written":9165216,"thread_start_us":705,"threads_started":14,"wal-append.queue_time_us":4776} I0107 12:49:06.174165 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 197619332 bytes on disk I0107 12:49:06.174343 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":30,"lbm_reads_lt_1ms":20} I0107 12:49:06.174456 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.383488 I0107 12:49:08.992033 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 38.891s user 10.218s sys 0.774s Metrics: {"bytes_written":157903836,"cfile_cache_hit":35,"cfile_cache_hit_bytes":217414,"cfile_cache_miss":21121,"cfile_cache_miss_bytes":655134826,"cfile_init":18,"delta_iterators_relevant":10,"dirs.queue_time_us":1194,"dirs.run_cpu_time_us":611,"dirs.run_wall_time_us":256641,"drs_written":5,"lbm_read_time_us":22346854,"lbm_reads_1-10_ms":239,"lbm_reads_10-100_ms":498,"lbm_reads_gt_100_ms":35,"lbm_reads_lt_1ms":20421,"lbm_write_time_us":2012968,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":31,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":15852,"num_input_rowsets":5,"rows_written":8978465,"spinlock_wait_cycles":896,"thread_start_us":819,"threads_started":14,"wal-append.queue_time_us":69} I0107 12:49:08.992244 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.397599 I0107 12:49:17.619563 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 35.708s user 10.074s sys 0.647s Metrics: {"bytes_written":161173046,"cfile_cache_hit":14157,"cfile_cache_hit_bytes":402794297,"cfile_cache_miss":7439,"cfile_cache_miss_bytes":267703987,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":369,"dirs.run_cpu_time_us":677,"dirs.run_wall_time_us":362106,"drs_written":5,"lbm_read_time_us":15605598,"lbm_reads_1-10_ms":113,"lbm_reads_10-100_ms":294,"lbm_reads_gt_100_ms":26,"lbm_reads_lt_1ms":7094,"lbm_write_time_us":5693630,"lbm_writes_1-10_ms":87,"lbm_writes_10-100_ms":112,"lbm_writes_gt_100_ms":8,"lbm_writes_lt_1ms":16029,"num_input_rowsets":6,"rows_written":9165141,"spinlock_wait_cycles":15744,"thread_start_us":194,"threads_started":6,"wal-append.queue_time_us":51} I0107 12:49:17.619693 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 197758664 bytes on disk I0107 12:49:17.619810 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":23,"lbm_reads_lt_1ms":20} I0107 12:49:17.619880 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling CompactRowSetsOp(16e58229a0c648eda40d4376454f2981): perf score=0.381748 I0107 12:49:28.967236 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 19.975s user 5.466s sys 0.419s Metrics: {"bytes_written":93114569,"cfile_cache_hit":4968,"cfile_cache_hit_bytes":141227684,"cfile_cache_miss":7516,"cfile_cache_miss_bytes":246548324,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":357,"dirs.run_cpu_time_us":379,"dirs.run_wall_time_us":184394,"drs_written":3,"lbm_read_time_us":11651552,"lbm_reads_1-10_ms":85,"lbm_reads_10-100_ms":244,"lbm_reads_gt_100_ms":15,"lbm_reads_lt_1ms":7220,"lbm_write_time_us":566454,"lbm_writes_1-10_ms":4,"lbm_writes_10-100_ms":5,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":9378,"num_input_rowsets":4,"rows_written":5298426,"thread_start_us":533,"threads_started":12,"wal-append.queue_time_us":68} I0107 12:49:28.967495 24837 maintenance_manager.cc:326] P 76f8bc397d684ec2b99494eb2fee53c4: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 281953299 bytes on disk I0107 12:49:29.633486 24731 maintenance_manager.cc:527] P 76f8bc397d684ec2b99494eb2fee53c4: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.666s user 0.001s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":665546,"lbm_reads_1-10_ms":6,"lbm_reads_10-100_ms":4,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":15} I0107 12:49:30.790796 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 24.616s user 5.966s sys 0.530s Metrics: {"bytes_written":93118818,"cfile_cache_hit":41,"cfile_cache_hit_bytes":3083272,"cfile_cache_miss":12439,"cfile_cache_miss_bytes":385017230,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":3300,"dirs.run_cpu_time_us":307,"dirs.run_wall_time_us":108586,"drs_written":3,"lbm_read_time_us":15319710,"lbm_reads_1-10_ms":188,"lbm_reads_10-100_ms":288,"lbm_reads_gt_100_ms":27,"lbm_reads_lt_1ms":11984,"lbm_write_time_us":600053,"lbm_writes_1-10_ms":10,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":9373,"num_input_rowsets":3,"rows_written":5298451,"spinlock_wait_cycles":7808,"thread_start_us":798,"threads_started":12,"wal-append.queue_time_us":58} I0107 12:49:30.790987 24588 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 114637387 bytes on disk I0107 12:49:30.909979 24482 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.119s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":118791,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":7} I0107 12:49:33.856323 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: CompactRowSetsOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 16.236s user 5.587s sys 0.361s Metrics: {"bytes_written":93117101,"cfile_cache_hit":8146,"cfile_cache_hit_bytes":234139135,"cfile_cache_miss":4340,"cfile_cache_miss_bytes":154342387,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":9184,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":66563,"drs_written":3,"lbm_read_time_us":4274182,"lbm_reads_1-10_ms":88,"lbm_reads_10-100_ms":99,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":4184,"lbm_write_time_us":4135815,"lbm_writes_1-10_ms":44,"lbm_writes_10-100_ms":86,"lbm_writes_gt_100_ms":5,"lbm_writes_lt_1ms":9253,"num_input_rowsets":3,"rows_written":5298447,"thread_start_us":556,"threads_started":11,"wal-append.queue_time_us":70} I0107 12:49:33.856539 24713 maintenance_manager.cc:326] P c0dcf07e91a6459fb83f90f61d363e44: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 114635315 bytes on disk I0107 12:49:33.917826 24607 maintenance_manager.cc:527] P c0dcf07e91a6459fb83f90f61d363e44: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.061s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":61069,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":6} I0107 12:49:44.043063 24384 linked_list-test-util.h:876] Done collecting results (14012700 rows in 64408.3ms) I0107 12:49:45.736922 24384 linked_list-test-util.h:667] Snapshot: P: 1578429764977164 usec, L: 1 verified. Result: OK I0107 12:49:45.738848 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578430064977359 usec, L: 1 Expected Rows: 19547200 I0107 12:49:45.813009 24384 linked_list-test-util.h:639] Calling callback on tserver 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:50:24.801760 24384 linked_list-test-util.h:876] Done collecting results (19547200 rows in 38988.8ms) I0107 12:50:27.269613 24384 linked_list-test-util.h:667] Snapshot: P: 1578430064977359 usec, L: 1 verified. Result: OK I0107 12:50:27.270273 24384 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 19547200 I0107 12:51:01.979933 24384 linked_list-test-util.h:876] Done collecting results (19547200 rows in 34161ms) I0107 12:51:04.434044 24384 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0107 12:51:04.434120 24384 linked_list-test-util.h:801] Successfully verified 19547200 rows I0107 12:51:04.438663 24384 linked_list-test.cc:194] Successfully verified 19547200 rows before killing any servers. I0107 12:51:04.440126 24384 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0107 12:51:04.440376 24384 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 914675d8717742d1ae5929637e95acb2 to finish bootstrapping I0107 12:51:04.441819 24384 ts_itest-base.cc:233] Waiting for 3 tablets on tserver c0dcf07e91a6459fb83f90f61d363e44 to finish bootstrapping I0107 12:51:04.443188 24384 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 76f8bc397d684ec2b99494eb2fee53c4 to finish bootstrapping I0107 12:51:04.444597 24384 linked_list-test.cc:199] Will restart the tablet server during verification scan. I0107 12:51:04.444926 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578429464976514 usec, L: 1 Expected Rows: 7530250 I0107 12:51:04.466337 24384 linked_list-test-util.h:639] Calling callback on tserver 914675d8717742d1ae5929637e95acb2 I0107 12:51:04.466406 24384 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24465 I0107 12:51:04.508450 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.208.1:35732 --local_ip_for_outbound_sockets=127.23.208.1 --webserver_port=39731 --webserver_interface=127.23.208.1 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:51:04.519830 30845 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:51:04.519929 30845 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:51:04.520741 30845 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:51:04.520771 30845 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.1 I0107 12:51:04.521796 30845 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.1:35732 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.23.208.1 --webserver_port=39731 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.30845 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:51:04.523941 30845 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:51:04.525118 30845 hybrid_clock.cc:186] HybridClock initialized: now 1578430264525107 us; error 16 us; skew 500 ppm I0107 12:51:04.525202 30845 fs_manager.cc:263] Metadata directory not provided I0107 12:51:04.525228 30845 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:51:04.527384 30845 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s I0107 12:51:04.527562 30845 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:51:04.527622 30845 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:51:04.529702 30845 fs_manager.cc:434] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s I0107 12:51:04.529873 30845 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal uuid: "914675d8717742d1ae5929637e95acb2" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:51:04.529907 30845 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/data Total live blocks: 66 Total live bytes: 766288035 Total live bytes (after alignment): 766431232 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:51:04.541622 30845 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:51:04.541817 30845 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0107 12:51:04.542603 30845 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0107 12:51:04.542623 30845 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0107 12:51:04.542920 30867 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:51:04.543162 30845 ts_tablet_manager.cc:415] Registered 3 tablets I0107 12:51:04.547876 30845 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.1:35732 I0107 12:51:04.548070 30845 webserver.cc:382] Webserver started at http://127.23.208.1:39731/ using document root and password file I0107 12:51:04.548195 30845 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb I0107 12:51:04.550735 30972 heartbeater.cc:322] Connected to a master server at 127.23.208.62:54889 I0107 12:51:04.550755 30972 heartbeater.cc:406] Registering TS with master... I0107 12:51:04.550804 30972 heartbeater.cc:455] Master 127.23.208.62:54889 requested a full tablet report, sending... I0107 12:51:04.550976 24414 ts_manager.cc:192] Re-registered known tserver with Master: 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732) I0107 12:51:04.551437 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 30845 I0107 12:51:04.551611 24414 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.208.1:53372 I0107 12:51:04.555440 30889 tablet_service.cc:2553] Scan: Not found: Scanner 24747a688bc144d89724b7a41303961a not found (it may have expired): call sequence id=1, remote={username='jenkins'} at 127.23.208.1:34454 W0107 12:51:04.555583 24384 client.cc:1692] Attempting to retry Scanner { table: TestTable, tablet: 16e58229a0c648eda40d4376454f2981, projection: ( rand_key INT64 NOT NULL, link_to INT64 NOT NULL, updated BOOL NOT NULL, PRIMARY KEY () ), scan_spec: } elsewhere. I0107 12:51:04.648372 30867 log.cc:815] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Log is configured to *not* fsync() on all Append() calls W0107 12:51:04.807271 24724 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 76f8bc397d684ec2b99494eb2fee53c4 -> Peer 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732): Couldn't send request to peer 914675d8717742d1ae5929637e95acb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry. W0107 12:51:04.826117 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732): Couldn't send request to peer 914675d8717742d1ae5929637e95acb2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry. I0107 12:51:04.947844 30867 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11240 overwritten=0 applied=11240 ignored=11240} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0107 12:51:04.947998 30867 log.cc:1202] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/77e67457a0334f92a22f52fe739203dd.recovery-1578430264947950 I0107 12:51:04.948045 30867 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Bootstrap complete. I0107 12:51:04.948107 30867 ts_tablet_manager.cc:1105] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Time spent bootstrapping tablet: real 0.405s user 0.293s sys 0.081s I0107 12:51:04.949024 30867 raft_consensus.cc:341] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:04.949082 30867 raft_consensus.cc:712] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Initialized, Role: FOLLOWER I0107 12:51:04.949184 30867 consensus_queue.cc:229] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 383367, Last appended: 1.383367, Last appended by leader: 383367, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:04.949353 30867 ts_tablet_manager.cc:1133] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0107 12:51:04.949420 30867 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:51:04.949365 30972 heartbeater.cc:447] Master 127.23.208.62:54889 was elected leader, sending a full tablet report... I0107 12:51:05.048772 30973 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd): 70340981 bytes on disk I0107 12:51:05.090337 30866 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(77e67457a0334f92a22f52fe739203dd) complete. Timing: real 0.041s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":41367,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0107 12:51:05.389976 30867 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10398 overwritten=0 applied=10398 ignored=10398} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0107 12:51:05.390065 30867 log.cc:1202] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/30019a15e66f4115b11a50dbcaf93a87.recovery-1578430265390022 I0107 12:51:05.390074 30867 tablet_bootstrap.cc:439] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Bootstrap complete. I0107 12:51:05.390103 30867 ts_tablet_manager.cc:1105] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Time spent bootstrapping tablet: real 0.441s user 0.341s sys 0.096s I0107 12:51:05.390194 30867 raft_consensus.cc:341] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.390223 30867 raft_consensus.cc:712] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Initialized, Role: FOLLOWER I0107 12:51:05.390275 30867 consensus_queue.cc:229] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 383563, Last appended: 1.383563, Last appended by leader: 383563, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.390329 30867 ts_tablet_manager.cc:1133] T 30019a15e66f4115b11a50dbcaf93a87 P 914675d8717742d1ae5929637e95acb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0107 12:51:05.390357 30867 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:51:05.488081 30867 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1578 overwritten=0 applied=1578 ignored=1578} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0107 12:51:05.488178 30867 log.cc:1202] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal/wals/16e58229a0c648eda40d4376454f2981.recovery-1578430265488132 I0107 12:51:05.488189 30867 tablet_bootstrap.cc:439] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Bootstrap complete. I0107 12:51:05.488222 30867 ts_tablet_manager.cc:1105] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Time spent bootstrapping tablet: real 0.098s user 0.085s sys 0.013s I0107 12:51:05.488327 30867 raft_consensus.cc:341] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.488343 30867 raft_consensus.cc:712] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Initialized, Role: FOLLOWER I0107 12:51:05.488385 30867 consensus_queue.cc:229] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 391813, Last appended: 1.391813, Last appended by leader: 391813, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.488457 30867 ts_tablet_manager.cc:1133] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I0107 12:51:05.590824 30973 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981): 281958166 bytes on disk I0107 12:51:05.625707 30866 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(16e58229a0c648eda40d4376454f2981) complete. Timing: real 0.035s user 0.000s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":34686,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":25} I0107 12:51:05.625782 30973 maintenance_manager.cc:326] P 914675d8717742d1ae5929637e95acb2: Scheduling UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87): 70413623 bytes on disk I0107 12:51:05.629837 30866 maintenance_manager.cc:527] P 914675d8717742d1ae5929637e95acb2: UndoDeltaBlockGCOp(30019a15e66f4115b11a50dbcaf93a87) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":3987,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":7} I0107 12:51:05.955315 30983 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 914675d8717742d1ae5929637e95acb2) I0107 12:51:05.955420 30983 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.955546 30983 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827) I0107 12:51:05.955670 30929 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 383367 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" is_pre_election: true I0107 12:51:05.955667 24668 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 383367 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" is_pre_election: true I0107 12:51:05.955718 30929 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76f8bc397d684ec2b99494eb2fee53c4 in term 1. I0107 12:51:05.955816 24724 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2; no voters: I0107 12:51:05.955860 30983 raft_consensus.cc:2696] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0107 12:51:05.955870 30983 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 914675d8717742d1ae5929637e95acb2) I0107 12:51:05.955888 30983 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 1 FOLLOWER]: Advancing to term 2 I0107 12:51:05.956513 30983 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.956557 30983 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 2 election: Requested vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827) I0107 12:51:05.956661 24668 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 383367 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" I0107 12:51:05.956682 30929 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "76f8bc397d684ec2b99494eb2fee53c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 383367 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" I0107 12:51:05.956696 30929 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Advancing to term 2 I0107 12:51:05.957731 30929 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76f8bc397d684ec2b99494eb2fee53c4 in term 2. I0107 12:51:05.957805 24724 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2; no voters: c0dcf07e91a6459fb83f90f61d363e44 I0107 12:51:05.957844 30983 raft_consensus.cc:2696] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 2 FOLLOWER]: Leader election won for term 2 I0107 12:51:05.957897 30983 raft_consensus.cc:672] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [term 2 LEADER]: Becoming Leader. State: Replica: 76f8bc397d684ec2b99494eb2fee53c4, State: Running, Role: LEADER I0107 12:51:05.957960 30983 consensus_queue.cc:206] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383367, Committed index: 383367, Last appended: 1.383367, Last appended by leader: 383367, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.958286 24414 catalog_manager.cc:4143] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 reported cstate change: term changed from 1 to 2, leader changed from 914675d8717742d1ae5929637e95acb2 (127.23.208.1) to 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3). New cstate: current_term: 2 leader_uuid: "76f8bc397d684ec2b99494eb2fee53c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } health_report { overall_health: HEALTHY } } } I0107 12:51:05.959197 30984 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 914675d8717742d1ae5929637e95acb2) I0107 12:51:05.959246 30984 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:51:05.959322 30984 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:51:05.959434 24793 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 383367 } } ignore_live_leader: false dest_uuid: "76f8bc397d684ec2b99494eb2fee53c4" is_pre_election: true I0107 12:51:05.959439 30929 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 383367 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" is_pre_election: true I0107 12:51:05.959484 30929 raft_consensus.cc:2321] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0dcf07e91a6459fb83f90f61d363e44 in current term 2: Already voted for candidate 76f8bc397d684ec2b99494eb2fee53c4 in this term. I0107 12:51:05.959565 24600 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2 I0107 12:51:05.959625 30984 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Advancing to term 2 I0107 12:51:05.960381 30984 raft_consensus.cc:2644] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0107 12:51:06.121949 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.306045 I0107 12:51:06.122792 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":435,"delete_count":0,"lbm_write_time_us":38,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1} I0107 12:51:06.387878 30929 raft_consensus.cc:1195] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Refusing update from remote peer 76f8bc397d684ec2b99494eb2fee53c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 383367. Preceding OpId from leader: term: 2 index: 383368. (index mismatch) I0107 12:51:06.388043 30983 consensus_queue.cc:982] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383368, Last known committed idx: 383367, Time since last communication: 0.000s I0107 12:51:06.388954 24668 raft_consensus.cc:1195] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Refusing update from remote peer 76f8bc397d684ec2b99494eb2fee53c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 383367. Preceding OpId from leader: term: 2 index: 383368. (index mismatch) I0107 12:51:06.389307 30997 consensus_queue.cc:982] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383368, Last known committed idx: 383367, Time since last communication: 0.000s I0107 12:51:24.667199 24384 linked_list-test-util.h:876] Done collecting results (7530250 rows in 20200.9ms) I0107 12:51:25.544407 24384 linked_list-test-util.h:667] Snapshot: P: 1578429464976514 usec, L: 1 verified. Result: OK I0107 12:51:25.545106 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578429764977164 usec, L: 1 Expected Rows: 14012700 I0107 12:51:25.546346 24384 linked_list-test-util.h:639] Calling callback on tserver 914675d8717742d1ae5929637e95acb2 I0107 12:52:02.056985 24384 linked_list-test-util.h:876] Done collecting results (14012700 rows in 36510.6ms) I0107 12:52:03.783416 24384 linked_list-test-util.h:667] Snapshot: P: 1578429764977164 usec, L: 1 verified. Result: OK I0107 12:52:03.783922 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578430064977359 usec, L: 1 Expected Rows: 19547200 I0107 12:52:03.835901 24384 linked_list-test-util.h:639] Calling callback on tserver c0dcf07e91a6459fb83f90f61d363e44 I0107 12:52:40.790786 24384 linked_list-test-util.h:876] Done collecting results (19547200 rows in 36954.9ms) I0107 12:52:43.220592 24384 linked_list-test-util.h:667] Snapshot: P: 1578430064977359 usec, L: 1 verified. Result: OK I0107 12:52:43.221122 24384 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 19547200 I0107 12:53:16.578845 24384 linked_list-test-util.h:876] Done collecting results (19547200 rows in 33169.6ms) I0107 12:53:19.046339 24384 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0107 12:53:19.046414 24384 linked_list-test-util.h:801] Successfully verified 19547200 rows I0107 12:53:19.046496 24384 linked_list-test.cc:204] Done with tserver restart test. I0107 12:53:19.051478 24384 linked_list-test.cc:210] Will kill the tablet server during verification scan. I0107 12:53:19.052012 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578429464976514 usec, L: 1 Expected Rows: 7530250 I0107 12:53:19.053390 24384 linked_list-test-util.h:639] Calling callback on tserver 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:53:19.053449 24384 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24715 I0107 12:53:19.102833 24384 meta_cache.cc:1038] Marking tablet server 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) as failed. W0107 12:53:19.102860 24384 meta_cache.cc:234] Tablet 16e58229a0c648eda40d4376454f2981: Replica 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) W0107 12:53:19.102880 24384 client.cc:1692] Attempting to retry Scanner { table: TestTable, tablet: 16e58229a0c648eda40d4376454f2981, projection: ( rand_key INT64 NOT NULL, link_to INT64 NOT NULL, updated BOOL NOT NULL, PRIMARY KEY () ), scan_spec: } elsewhere. W0107 12:53:19.489964 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0107 12:53:20.224908 31175 raft_consensus.cc:477] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76f8bc397d684ec2b99494eb2fee53c4) I0107 12:53:20.224980 31175 raft_consensus.cc:499] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.225082 31175 leader_election.cc:293] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) W0107 12:53:20.225261 24600 leader_election.cc:339] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) I0107 12:53:20.225338 30929 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 2 candidate_status { last_received { term: 1 index: 391813 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" is_pre_election: true I0107 12:53:20.225471 24600 leader_election.cc:307] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2 I0107 12:53:20.225535 31175 raft_consensus.cc:2644] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0107 12:53:20.244621 31175 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 76f8bc397d684ec2b99494eb2fee53c4) I0107 12:53:20.244702 31175 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.244763 31175 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) W0107 12:53:20.244946 24600 leader_election.cc:339] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) I0107 12:53:20.245012 30929 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "c0dcf07e91a6459fb83f90f61d363e44" candidate_term: 3 candidate_status { last_received { term: 2 index: 383368 } } ignore_live_leader: false dest_uuid: "914675d8717742d1ae5929637e95acb2" is_pre_election: true I0107 12:53:20.245100 24600 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4, 914675d8717742d1ae5929637e95acb2 I0107 12:53:20.245147 31175 raft_consensus.cc:2644] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0107 12:53:20.327181 31177 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 76f8bc397d684ec2b99494eb2fee53c4) I0107 12:53:20.327257 31177 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.327381 31177 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) W0107 12:53:20.327689 30859 leader_election.cc:339] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) I0107 12:53:20.332209 24668 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 3 candidate_status { last_received { term: 2 index: 383368 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" is_pre_election: true I0107 12:53:20.332237 24668 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 2. I0107 12:53:20.332317 30860 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:53:20.332378 31177 raft_consensus.cc:2696] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0107 12:53:20.332392 31177 raft_consensus.cc:477] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 76f8bc397d684ec2b99494eb2fee53c4) I0107 12:53:20.332399 31177 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Advancing to term 3 I0107 12:53:20.332953 31177 raft_consensus.cc:499] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.333040 31177 leader_election.cc:293] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 3 election: Requested vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) I0107 12:53:20.333073 24668 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "77e67457a0334f92a22f52fe739203dd" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 3 candidate_status { last_received { term: 2 index: 383368 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" I0107 12:53:20.333092 24668 raft_consensus.cc:2949] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Advancing to term 3 W0107 12:53:20.333160 30859 leader_election.cc:339] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) I0107 12:53:20.333899 24668 raft_consensus.cc:2396] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 3. I0107 12:53:20.333986 30860 leader_election.cc:307] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:53:20.334029 31177 raft_consensus.cc:2696] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 3 FOLLOWER]: Leader election won for term 3 I0107 12:53:20.334137 31177 raft_consensus.cc:672] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [term 3 LEADER]: Becoming Leader. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Running, Role: LEADER I0107 12:53:20.334220 31177 consensus_queue.cc:206] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383368, Committed index: 383368, Last appended: 2.383368, Last appended by leader: 383368, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.334645 24420 catalog_manager.cc:4143] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 reported cstate change: term changed from 2 to 3, leader changed from 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3) to 914675d8717742d1ae5929637e95acb2 (127.23.208.1). New cstate: current_term: 3 leader_uuid: "914675d8717742d1ae5929637e95acb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } health_report { overall_health: UNKNOWN } } } I0107 12:53:20.467309 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.037318 I0107 12:53:20.468071 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":434,"delete_count":0,"lbm_write_time_us":54,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1} I0107 12:53:20.468122 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.281868 I0107 12:53:20.468922 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_cache_hit":5,"cfile_cache_hit_bytes":2949,"cfile_cache_miss":2,"cfile_cache_miss_bytes":333,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":100,"lbm_read_time_us":12,"lbm_reads_lt_1ms":10,"lbm_write_time_us":29,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":126,"threads_started":2,"update_count":2} I0107 12:53:20.468982 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000 I0107 12:53:20.469048 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0107 12:53:20.531698 31177 raft_consensus.cc:477] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76f8bc397d684ec2b99494eb2fee53c4) I0107 12:53:20.531774 31177 raft_consensus.cc:499] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.531831 31177 leader_election.cc:293] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) W0107 12:53:20.532019 30859 leader_election.cc:339] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) I0107 12:53:20.532074 24668 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 391813 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" is_pre_election: true I0107 12:53:20.532101 24668 raft_consensus.cc:2396] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 1. I0107 12:53:20.532199 30860 leader_election.cc:307] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:53:20.532255 31177 raft_consensus.cc:2696] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0107 12:53:20.532271 31177 raft_consensus.cc:477] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76f8bc397d684ec2b99494eb2fee53c4) I0107 12:53:20.532277 31177 raft_consensus.cc:2949] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 1 FOLLOWER]: Advancing to term 2 I0107 12:53:20.533073 31177 raft_consensus.cc:499] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.533115 31177 leader_election.cc:293] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 2 election: Requested vote from peers c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827), 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) W0107 12:53:20.533255 30859 leader_election.cc:339] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) I0107 12:53:20.533268 24668 tablet_service.cc:1307] Received RequestConsensusVote() RPC: tablet_id: "16e58229a0c648eda40d4376454f2981" candidate_uuid: "914675d8717742d1ae5929637e95acb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 391813 } } ignore_live_leader: false dest_uuid: "c0dcf07e91a6459fb83f90f61d363e44" I0107 12:53:20.533280 24668 raft_consensus.cc:2949] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 1 FOLLOWER]: Advancing to term 2 I0107 12:53:20.533972 24668 raft_consensus.cc:2396] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914675d8717742d1ae5929637e95acb2 in term 2. I0107 12:53:20.534075 30860 leader_election.cc:307] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914675d8717742d1ae5929637e95acb2, c0dcf07e91a6459fb83f90f61d363e44; no voters: 76f8bc397d684ec2b99494eb2fee53c4 I0107 12:53:20.534119 31177 raft_consensus.cc:2696] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 2 FOLLOWER]: Leader election won for term 2 I0107 12:53:20.534140 31177 raft_consensus.cc:672] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [term 2 LEADER]: Becoming Leader. State: Replica: 914675d8717742d1ae5929637e95acb2, State: Running, Role: LEADER I0107 12:53:20.534188 31177 consensus_queue.cc:206] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391813, Committed index: 391813, Last appended: 1.391813, Last appended by leader: 391813, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } } I0107 12:53:20.534591 24420 catalog_manager.cc:4143] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 reported cstate change: term changed from 1 to 2, leader changed from 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3) to 914675d8717742d1ae5929637e95acb2 (127.23.208.1). New cstate: current_term: 2 leader_uuid: "914675d8717742d1ae5929637e95acb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914675d8717742d1ae5929637e95acb2" member_type: VOTER last_known_addr { host: "127.23.208.1" port: 35732 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76f8bc397d684ec2b99494eb2fee53c4" member_type: VOTER last_known_addr { host: "127.23.208.3" port: 49994 } health_report { overall_health: UNKNOWN } } } I0107 12:53:20.864711 24668 raft_consensus.cc:1195] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44 [term 3 FOLLOWER]: Refusing update from remote peer 914675d8717742d1ae5929637e95acb2: Log matching property violated. Preceding OpId in replica: term: 2 index: 383368. Preceding OpId from leader: term: 3 index: 383369. (index mismatch) I0107 12:53:20.864869 31177 consensus_queue.cc:982] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383369, Last known committed idx: 383368, Time since last communication: 0.000s W0107 12:53:20.865769 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0107 12:53:20.935647 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0107 12:53:21.065153 24668 raft_consensus.cc:1195] T 16e58229a0c648eda40d4376454f2981 P c0dcf07e91a6459fb83f90f61d363e44 [term 2 FOLLOWER]: Refusing update from remote peer 914675d8717742d1ae5929637e95acb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 391813. Preceding OpId from leader: term: 2 index: 391814. (index mismatch) I0107 12:53:21.065364 31181 consensus_queue.cc:982] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0dcf07e91a6459fb83f90f61d363e44" member_type: VOTER last_known_addr { host: "127.23.208.2" port: 57827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391814, Last known committed idx: 391813, Time since last communication: 0.000s W0107 12:53:21.962491 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0107 12:53:23.400292 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0107 12:53:23.465355 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0107 12:53:24.395200 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0107 12:53:25.899875 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0107 12:53:26.122301 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0107 12:53:26.769831 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0107 12:53:28.327111 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0107 12:53:28.653942 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0107 12:53:29.215528 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0107 12:53:30.759711 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0107 12:53:31.037963 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0107 12:53:31.686549 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. I0107 12:53:32.590978 24384 meta_cache.cc:1038] Marking tablet server 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) as failed. W0107 12:53:32.590999 24384 meta_cache.cc:234] Tablet 16e58229a0c648eda40d4376454f2981: Replica 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) W0107 12:53:33.333724 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0107 12:53:33.572795 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0107 12:53:34.560448 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0107 12:53:35.896199 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0107 12:53:36.248436 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0107 12:53:37.073556 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0107 12:53:38.257535 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0107 12:53:38.570111 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0107 12:53:39.521355 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. I0107 12:53:39.585248 24384 linked_list-test-util.h:876] Done collecting results (7530250 rows in 20531.9ms) I0107 12:53:40.472808 24384 linked_list-test-util.h:667] Snapshot: P: 1578429464976514 usec, L: 1 verified. Result: OK I0107 12:53:40.473347 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578429764977164 usec, L: 1 Expected Rows: 14012700 I0107 12:53:40.474668 24384 linked_list-test-util.h:639] Calling callback on tserver 914675d8717742d1ae5929637e95acb2 W0107 12:53:40.731956 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0107 12:53:41.397717 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0107 12:53:42.140878 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0107 12:53:43.248531 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0107 12:53:43.968055 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0107 12:53:44.667013 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0107 12:53:45.862210 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0107 12:53:46.354370 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0107 12:53:47.149268 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0107 12:53:48.596894 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0107 12:53:48.831063 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0107 12:53:49.938604 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0107 12:53:50.812525 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0107 12:53:51.178869 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0107 12:53:52.495384 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0107 12:53:53.538071 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0107 12:53:53.667901 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0107 12:53:54.958114 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0107 12:53:56.154325 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0107 12:53:56.306139 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0107 12:53:57.376705 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0107 12:53:58.820816 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0107 12:53:58.924688 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0107 12:53:59.725435 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0107 12:54:01.347373 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0107 12:54:01.411325 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0107 12:54:02.153172 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0107 12:54:03.827858 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0107 12:54:04.067414 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0107 12:54:04.662488 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0107 12:54:06.195503 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0107 12:54:06.695529 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0107 12:54:07.293987 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0107 12:54:08.788978 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0107 12:54:09.297439 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0107 12:54:09.805187 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0107 12:54:11.324854 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0107 12:54:11.868302 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0107 12:54:12.480372 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0107 12:54:14.197918 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0107 12:54:14.400600 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0107 12:54:14.957506 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. W0107 12:54:16.648907 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. I0107 12:54:16.713747 24384 linked_list-test-util.h:876] Done collecting results (14012700 rows in 36239.1ms) W0107 12:54:17.120658 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. W0107 12:54:17.163691 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. I0107 12:54:18.437146 24384 linked_list-test-util.h:667] Snapshot: P: 1578429764977164 usec, L: 1 verified. Result: OK I0107 12:54:18.437669 24384 linked_list-test-util.h:620] Verifying Snapshot: P: 1578430064977359 usec, L: 1 Expected Rows: 19547200 I0107 12:54:18.438663 24384 linked_list-test-util.h:639] Calling callback on tserver 914675d8717742d1ae5929637e95acb2 W0107 12:54:18.693528 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0107 12:54:19.770802 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0107 12:54:19.970475 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0107 12:54:21.089617 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0107 12:54:22.386404 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0107 12:54:22.472316 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0107 12:54:23.670255 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0107 12:54:24.788206 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0107 12:54:24.981922 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0107 12:54:26.082028 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0107 12:54:27.367197 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0107 12:54:27.408135 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0107 12:54:28.708568 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0107 12:54:29.730655 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0107 12:54:29.907929 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0107 12:54:31.417085 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0107 12:54:32.148496 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0107 12:54:32.418161 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0107 12:54:33.827188 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0107 12:54:34.618868 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0107 12:54:35.186991 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0107 12:54:36.354509 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0107 12:54:37.263056 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0107 12:54:37.699156 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0107 12:54:38.712496 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0107 12:54:40.069797 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0107 12:54:40.110719 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0107 12:54:41.255298 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0107 12:54:42.595485 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0107 12:54:42.615743 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. I0107 12:54:42.616863 24384 meta_cache.cc:1038] Marking tablet server 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) as failed. W0107 12:54:42.616883 24384 meta_cache.cc:234] Tablet 16e58229a0c648eda40d4376454f2981: Replica 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111) W0107 12:54:43.796581 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0107 12:54:44.979614 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0107 12:54:45.119163 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0107 12:54:46.149451 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0107 12:54:47.662371 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0107 12:54:47.847420 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0107 12:54:48.997486 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0107 12:54:50.304486 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0107 12:54:50.348520 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0107 12:54:51.219522 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0107 12:54:52.756698 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0107 12:54:52.885778 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0107 12:54:53.913887 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. I0107 12:54:54.865947 24384 linked_list-test-util.h:876] Done collecting results (19547200 rows in 36427.3ms) W0107 12:54:55.048753 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0107 12:54:55.335439 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0107 12:54:56.306541 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. I0107 12:54:57.299693 24384 linked_list-test-util.h:667] Snapshot: P: 1578430064977359 usec, L: 1 verified. Result: OK I0107 12:54:57.300297 24384 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 19547200 W0107 12:54:57.747577 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0107 12:54:57.994676 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0107 12:54:59.025558 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0107 12:55:00.204169 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0107 12:55:00.359086 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0107 12:55:01.525202 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0107 12:55:02.495685 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0107 12:55:02.652618 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0107 12:55:03.916245 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0107 12:55:04.656723 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0107 12:55:05.205853 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0107 12:55:06.328814 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0107 12:55:07.307695 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0107 12:55:07.815462 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0107 12:55:09.004415 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0107 12:55:10.077822 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0107 12:55:10.351305 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0107 12:55:11.625710 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0107 12:55:12.410990 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0107 12:55:12.686586 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0107 12:55:14.040683 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0107 12:55:14.920845 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0107 12:55:15.253803 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0107 12:55:16.588383 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0107 12:55:17.249014 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0107 12:55:17.636333 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0107 12:55:19.033962 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0107 12:55:19.958763 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0107 12:55:20.039857 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. I0107 12:55:20.554253 24411 maintenance_manager.cc:326] P c12eb4d7452f4af5abdaa23e9a3d5e7e: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033357 I0107 12:55:20.555128 24410 maintenance_manager.cc:527] P c12eb4d7452f4af5abdaa23e9a3d5e7e: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":434,"delete_count":0,"lbm_write_time_us":34,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":29,"threads_started":1,"update_count":1,"wal-append.queue_time_us":52} W0107 12:55:21.863752 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0107 12:55:22.244063 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0107 12:55:22.633464 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0107 12:55:24.127346 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0107 12:55:24.708688 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0107 12:55:25.134603 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0107 12:55:26.887789 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0107 12:55:27.335916 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0107 12:55:27.382900 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0107 12:55:29.655390 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0107 12:55:29.685119 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0107 12:55:30.102998 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0107 12:55:32.228509 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0107 12:55:32.668017 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry. W0107 12:55:32.837509 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. I0107 12:55:33.103292 24384 linked_list-test-util.h:876] Done collecting results (19547200 rows in 35548.6ms) W0107 12:55:34.737285 30859 consensus_peers.cc:473] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry. W0107 12:55:35.136155 24600 consensus_peers.cc:473] T 30019a15e66f4115b11a50dbcaf93a87 P c0dcf07e91a6459fb83f90f61d363e44 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry. W0107 12:55:35.256202 30859 consensus_peers.cc:473] T 16e58229a0c648eda40d4376454f2981 P 914675d8717742d1ae5929637e95acb2 -> Peer 76f8bc397d684ec2b99494eb2fee53c4 (127.23.208.3:49994): Couldn't send request to peer 76f8bc397d684ec2b99494eb2fee53c4. Status: Network error: Client connection negotiation failed: client connection to 127.23.208.3:49994: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry. I0107 12:55:35.514024 24384 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0107 12:55:35.514096 24384 linked_list-test-util.h:801] Successfully verified 19547200 rows I0107 12:55:35.514117 24384 linked_list-test.cc:215] Done with tserver kill test. I0107 12:55:35.517554 24384 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 30845 I0107 12:55:35.556385 24384 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24591 I0107 12:55:35.594329 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.208.1:35732 --local_ip_for_outbound_sockets=127.23.208.1 --webserver_port=39731 --webserver_interface=127.23.208.1 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:55:35.605846 31501 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:55:35.605952 31501 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:55:35.606712 31501 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:55:35.606741 31501 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.1 I0107 12:55:35.607774 31501 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.1:35732 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.23.208.1 --webserver_port=39731 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.31501 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:55:35.609939 31501 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:55:35.611119 31501 hybrid_clock.cc:186] HybridClock initialized: now 1578430535611110 us; error 13 us; skew 500 ppm I0107 12:55:35.611207 31501 fs_manager.cc:263] Metadata directory not provided I0107 12:55:35.611230 31501 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:55:35.613433 31501 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s I0107 12:55:35.613651 31501 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:55:35.613714 31501 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:55:35.615797 31501 fs_manager.cc:434] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0107 12:55:35.615981 31501 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal uuid: "914675d8717742d1ae5929637e95acb2" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:55:35.616015 31501 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/data Total live blocks: 66 Total live bytes: 766288035 Total live bytes (after alignment): 766431232 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:55:35.643939 31501 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:55:35.644160 31501 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0107 12:55:35.644960 31501 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0107 12:55:35.644979 31501 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0107 12:55:35.645272 31523 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Bootstrap starting. I0107 12:55:35.645547 31501 ts_tablet_manager.cc:415] Registered 3 tablets I0107 12:55:35.653723 31501 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.1:35732 I0107 12:55:35.653959 31501 webserver.cc:382] Webserver started at http://127.23.208.1:39731/ using document root and password file I0107 12:55:35.654093 31501 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-0/data/info.pb I0107 12:55:35.657106 31628 heartbeater.cc:322] Connected to a master server at 127.23.208.62:54889 I0107 12:55:35.657124 31628 heartbeater.cc:406] Registering TS with master... I0107 12:55:35.657156 31628 heartbeater.cc:455] Master 127.23.208.62:54889 requested a full tablet report, sending... I0107 12:55:35.657336 24420 ts_manager.cc:192] Re-registered known tserver with Master: 914675d8717742d1ae5929637e95acb2 (127.23.208.1:35732) I0107 12:55:35.657568 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 31501 I0107 12:55:35.657727 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.208.2:57827 --local_ip_for_outbound_sockets=127.23.208.2 --webserver_port=48525 --webserver_interface=127.23.208.2 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0107 12:55:35.658177 24420 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.208.1:59395 WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:55:35.669271 31632 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:55:35.669373 31632 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:55:35.670140 31632 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:55:35.670169 31632 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.2 I0107 12:55:35.671205 31632 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.2:57827 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/info.pb --webserver_interface=127.23.208.2 --webserver_port=48525 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.31632 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.2 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:55:35.673352 31632 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:55:35.674476 31632 hybrid_clock.cc:186] HybridClock initialized: now 1578430535674468 us; error 17 us; skew 500 ppm I0107 12:55:35.674568 31632 fs_manager.cc:263] Metadata directory not provided I0107 12:55:35.674593 31632 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:55:35.676729 31632 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s I0107 12:55:35.676879 31632 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:55:35.676940 31632 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:55:35.679342 31632 fs_manager.cc:434] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0107 12:55:35.679543 31632 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal uuid: "c0dcf07e91a6459fb83f90f61d363e44" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:55:35.679576 31632 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/data Total live blocks: 66 Total live bytes: 766282852 Total live bytes (after alignment): 766423040 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:55:35.727039 31523 log.cc:815] T 77e67457a0334f92a22f52fe739203dd P 914675d8717742d1ae5929637e95acb2: Log is configured to *not* fsync() on all Append() calls I0107 12:55:35.745435 31632 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:55:35.745647 31632 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0107 12:55:35.746433 31632 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0107 12:55:35.746450 31632 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0107 12:55:35.746732 31656 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P c0dcf07e91a6459fb83f90f61d363e44: Bootstrap starting. I0107 12:55:35.747167 31632 ts_tablet_manager.cc:415] Registered 3 tablets I0107 12:55:35.757791 31632 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.2:57827 I0107 12:55:35.758067 31632 webserver.cc:382] Webserver started at http://127.23.208.2:48525/ using document root and password file I0107 12:55:35.758195 31632 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-1/data/info.pb I0107 12:55:35.761862 31761 heartbeater.cc:322] Connected to a master server at 127.23.208.62:54889 I0107 12:55:35.761886 31761 heartbeater.cc:406] Registering TS with master... I0107 12:55:35.761930 31761 heartbeater.cc:455] Master 127.23.208.62:54889 requested a full tablet report, sending... I0107 12:55:35.762073 24384 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 31632 I0107 12:55:35.762315 24384 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.208.3:49994 --local_ip_for_outbound_sockets=127.23.208.3 --webserver_port=46425 --webserver_interface=127.23.208.3 --tserver_master_addrs=127.23.208.62:54889 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.208.20:47721 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0107 12:55:35.763999 24420 ts_manager.cc:192] Re-registered known tserver with Master: c0dcf07e91a6459fb83f90f61d363e44 (127.23.208.2:57827) I0107 12:55:35.764804 24420 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.208.2:50712 WARNING: Logging before InitGoogleLogging() is written to STDERR W0107 12:55:35.779915 31764 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0107 12:55:35.780045 31764 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0107 12:55:35.781194 31764 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0107 12:55:35.781229 31764 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.208.3 I0107 12:55:35.782711 31764 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.208.20:47721 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.208.3:49994 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/info.pb --webserver_interface=127.23.208.3 --webserver_port=46425 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.208.62:54889 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.31764 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.208.3 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 50a4b763f09ff6a4a8955dd213e0e37a94e38b92 build type RELEASE built by jenkins at 07 Jan 2020 12:19:58 PST on quasar-wmlcin-1.vpc.cloudera.com I0107 12:55:35.785908 31764 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0107 12:55:35.787165 31764 hybrid_clock.cc:186] HybridClock initialized: now 1578430535787153 us; error 17 us; skew 500 ppm I0107 12:55:35.787302 31764 fs_manager.cc:263] Metadata directory not provided I0107 12:55:35.787360 31764 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0107 12:55:35.790732 31764 fs_manager.cc:414] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s I0107 12:55:35.790971 31764 env_posix.cc:1680] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0107 12:55:35.791074 31764 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0107 12:55:35.794737 31764 fs_manager.cc:434] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0107 12:55:35.794948 31764 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal uuid: "76f8bc397d684ec2b99494eb2fee53c4" format_stamp: "Formatted at 2020-01-07 20:32:44 on quasar-wmlcin-1.vpc.cloudera.com" I0107 12:55:35.794979 31764 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578429164263051-24384/linked-list-cluster/ts-2/data/data Total live blocks: 66 Total live bytes: 766279657 Total live bytes (after alignment): 766423040 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0107 12:55:35.816540 31764 env_posix.cc:1675] Raising this process' running threads per effective uid limit from 32768 to 65536 I0107 12:55:35.816787 31764 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0107 12:55:35.818006 31764 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0107 12:55:35.818028 31764 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0107 12:55:35.818447 31787 tablet_bootstrap.cc:439] T 77e67457a0334f92a22f52fe739203dd P 76f8bc397d684ec2b99494eb2fee53c4: Bootstrap starting. I0107 12:55:35.818890 31764 ts_tablet_manager.cc:415] Registered 3 tablets I0107 12:55:35.832968 31764 rpc_server.cc:211] RPC server started. Bound to: 127.23.208.3:49994 E0107 12:55:35.833159 31764 webserver.cc:443] Webserver: set_ports_option: cannot bind to 127.23.208.3:46425: 98 (Address already in use) W0107 12:55:35.833191 31764 net_util.cc:463] Trying to use lsof to find any processes listening on 0.0.0.0:46425 I0107 12:55:35.833197 31764 net_util.cc:466] $ export PATH=$PATH:/usr/sbin ; lsof -n -i 'TCP:46425' -sTCP:LISTEN ; for pid in $(lsof -F p -n -i 'TCP:46425' -sTCP:LISTEN | grep p | cut -f 2 -dp) ; do while [ $pid -gt 1 ] ; do ps h -fp $pid ; stat=($(