[==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from LinkedListTest [ RUN ] LinkedListTest.TestLoadAndVerify WARNING: Logging before InitGoogleLogging() is written to STDERR I0108 18:39:20.438849 24709 test_util.cc:229] Using random seed: -359794629 I0108 18:39:20.442878 24709 linked_list-test.cc:100] Linked List Test Configuration: I0108 18:39:20.442893 24709 linked_list-test.cc:101] -------------- I0108 18:39:20.442901 24709 linked_list-test.cc:102] 50 chains I0108 18:39:20.442905 24709 linked_list-test.cc:103] 3 tablets I0108 18:39:20.442909 24709 linked_list-test.cc:104] Mutations on I0108 18:39:20.442911 24709 linked_list-test.cc:105] -------------- I0108 18:39:20.442939 24709 ts_itest-base.cc:112] Starting cluster with: I0108 18:39:20.442943 24709 ts_itest-base.cc:113] -------------- I0108 18:39:20.442947 24709 ts_itest-base.cc:114] 3 tablet servers I0108 18:39:20.442950 24709 ts_itest-base.cc:115] 3 replicas per TS I0108 18:39:20.442953 24709 ts_itest-base.cc:116] -------------- 2020-01-09T02:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-01-09T02:39:20Z Disabled control of system clock Could not open connection to daemon I0108 18:39:20.447826 24709 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.24.33.126:55767 --webserver_interface=127.24.33.126 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true --builtin_ntp_servers=127.24.33.84:49727 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 18:39:20.458726 24719 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 18:39:20.458835 24719 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 18:39:20.459738 24719 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0108 18:39:20.459753 24719 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 18:39:20.459755 24719 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0108 18:39:20.459758 24719 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0108 18:39:20.460956 24719 master_main.cc:96] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.24.33.84:49727 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.24.33.126:55767 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/info.pb --webserver_interface=127.24.33.126 --webserver_port=0 --tablet_history_max_age_sec=86400 --never_fsync=true --heap_profile_path=/tmp/kudu-master.24719 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 4dca6b02a5e7b2e589646e79bc6e509f27c13c3d build type RELEASE built by jenkins at 08 Jan 2020 18:26:25 PST on quasar-luegrb-1.vpc.cloudera.com I0108 18:39:20.462955 24719 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 18:39:20.464098 24719 hybrid_clock.cc:186] HybridClock initialized: now 1578537560464080 us; error 31 us; skew 500 ppm I0108 18:39:20.464160 24719 fs_manager.cc:263] Metadata directory not provided I0108 18:39:20.464175 24719 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 18:39:20.464217 24719 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 18:39:20.464226 24719 server_base.cc:453] Attempting to create new FS layout instead I0108 18:39:20.464875 24719 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/instance: uuid: "86570b464d814d0eb2725157358e3a2f" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.464987 24719 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/wal/instance: uuid: "86570b464d814d0eb2725157358e3a2f" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.465792 24719 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/data/block_manager_instance: No such file or directory (error 2) I0108 18:39:20.476985 24719 fs_manager.cc:518] Time spent creating directory manager: real 0.012s user 0.000s sys 0.003s I0108 18:39:20.477113 24719 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 18:39:20.477169 24719 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 18:39:20.478077 24719 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0108 18:39:20.478102 24719 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/wal uuid: "86570b464d814d0eb2725157358e3a2f" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.478124 24719 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 18:39:20.530467 24719 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 18:39:20.533114 24719 rpc_server.cc:211] RPC server started. Bound to: 127.24.33.126:55767 I0108 18:39:20.533334 24719 webserver.cc:382] Webserver started at http://127.24.33.126:54254/ using document root and password file I0108 18:39:20.533463 24719 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/master-0/data/info.pb I0108 18:39:20.533691 24780 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.535419 24780 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f: Bootstrap starting. I0108 18:39:20.535630 24780 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f: No blocks or log segments found. Creating new log. I0108 18:39:20.535740 24780 log.cc:815] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f: Log is configured to *not* fsync() on all Append() calls I0108 18:39:20.536831 24780 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f: No bootstrap required, opened a new log I0108 18:39:20.537904 24780 raft_consensus.cc:341] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } I0108 18:39:20.537922 24780 raft_consensus.cc:367] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.537930 24780 raft_consensus.cc:712] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86570b464d814d0eb2725157358e3a2f, State: Initialized, Role: FOLLOWER I0108 18:39:20.537969 24780 consensus_queue.cc:229] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } I0108 18:39:20.537982 24780 raft_consensus.cc:381] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0108 18:39:20.537989 24780 raft_consensus.cc:477] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0108 18:39:20.537995 24780 raft_consensus.cc:2949] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.538275 24780 raft_consensus.cc:499] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } I0108 18:39:20.538305 24780 leader_election.cc:307] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86570b464d814d0eb2725157358e3a2f; no voters: I0108 18:39:20.538388 24780 leader_election.cc:293] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [CANDIDATE]: Term 1 election: Requested vote from peers I0108 18:39:20.538450 24782 raft_consensus.cc:2696] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 1 FOLLOWER]: Leader election won for term 1 I0108 18:39:20.538580 24782 raft_consensus.cc:672] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [term 1 LEADER]: Becoming Leader. State: Replica: 86570b464d814d0eb2725157358e3a2f, State: Running, Role: LEADER I0108 18:39:20.538652 24782 consensus_queue.cc:206] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } I0108 18:39:20.538818 24780 sys_catalog.cc:340] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "86570b464d814d0eb2725157358e3a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } } I0108 18:39:20.538834 24784 sys_catalog.cc:340] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86570b464d814d0eb2725157358e3a2f. Latest consensus state: current_term: 1 leader_uuid: "86570b464d814d0eb2725157358e3a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } } I0108 18:39:20.538854 24783 sys_catalog.cc:340] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86570b464d814d0eb2725157358e3a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86570b464d814d0eb2725157358e3a2f" member_type: VOTER } } I0108 18:39:20.538873 24780 sys_catalog.cc:343] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: This master's current role is: LEADER I0108 18:39:20.538928 24784 sys_catalog.cc:343] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: This master's current role is: LEADER I0108 18:39:20.538949 24783 sys_catalog.cc:343] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: This master's current role is: LEADER I0108 18:39:20.539173 24780 sys_catalog.cc:440] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f [sys.catalog]: configured and running, proceeding with master startup. I0108 18:39:20.539273 24787 catalog_manager.cc:1089] Loading table and tablet metadata into memory... I0108 18:39:20.539528 24787 catalog_manager.cc:1100] Initializing Kudu internal certificate authority... I0108 18:39:20.539932 24709 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master as pid 24719 I0108 18:39:20.540154 24709 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.24.33.65:0 --local_ip_for_outbound_sockets=127.24.33.65 --webserver_interface=127.24.33.65 --webserver_port=0 --tserver_master_addrs=127.24.33.126:55767 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.24.33.84:49727 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 18:39:20.553489 24790 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 18:39:20.553707 24790 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 18:39:20.554814 24790 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 18:39:20.554879 24790 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.33.65 I0108 18:39:20.556202 24790 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.24.33.84:49727 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.24.33.65:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.24.33.65 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.24.33.126:55767 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24790 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.24.33.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 4dca6b02a5e7b2e589646e79bc6e509f27c13c3d build type RELEASE built by jenkins at 08 Jan 2020 18:26:25 PST on quasar-luegrb-1.vpc.cloudera.com I0108 18:39:20.559342 24790 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 18:39:20.560530 24790 hybrid_clock.cc:186] HybridClock initialized: now 1578537560560518 us; error 19 us; skew 500 ppm I0108 18:39:20.560660 24790 fs_manager.cc:263] Metadata directory not provided I0108 18:39:20.560712 24790 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 18:39:20.560803 24790 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 18:39:20.560839 24790 server_base.cc:453] Attempting to create new FS layout instead I0108 18:39:20.561775 24790 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/instance: uuid: "c6ad452ff3b64d949298a50527e1fbb0" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.561950 24790 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/instance: uuid: "c6ad452ff3b64d949298a50527e1fbb0" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.563273 24790 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/data/block_manager_instance: No such file or directory (error 2) I0108 18:39:20.567400 24790 fs_manager.cc:518] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s I0108 18:39:20.567605 24790 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 18:39:20.567713 24790 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 18:39:20.569103 24790 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s I0108 18:39:20.569162 24790 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal uuid: "c6ad452ff3b64d949298a50527e1fbb0" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.569226 24790 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 18:39:20.601756 24787 catalog_manager.cc:984] Generated new certificate authority record I0108 18:39:20.602658 24787 catalog_manager.cc:1109] Loading token signing keys... I0108 18:39:20.609747 24787 catalog_manager.cc:4455] T 00000000000000000000000000000000 P 86570b464d814d0eb2725157358e3a2f: Generated new TSK 0 I0108 18:39:20.609841 24787 catalog_manager.cc:1119] Initializing in-progress tserver states... I0108 18:39:20.666460 24790 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 18:39:20.666656 24790 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0108 18:39:20.666668 24790 ts_tablet_manager.cc:415] Registered 0 tablets I0108 18:39:20.671882 24790 rpc_server.cc:211] RPC server started. Bound to: 127.24.33.65:55934 I0108 18:39:20.672109 24790 webserver.cc:382] Webserver started at http://127.24.33.65:35659/ using document root and password file I0108 18:39:20.672277 24790 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/info.pb I0108 18:39:20.673290 24709 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24790 I0108 18:39:20.673517 24709 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.24.33.66:0 --local_ip_for_outbound_sockets=127.24.33.66 --webserver_interface=127.24.33.66 --webserver_port=0 --tserver_master_addrs=127.24.33.126:55767 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.24.33.84:49727 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0108 18:39:20.675717 24912 heartbeater.cc:322] Connected to a master server at 127.24.33.126:55767 I0108 18:39:20.675737 24912 heartbeater.cc:406] Registering TS with master... I0108 18:39:20.675779 24912 heartbeater.cc:455] Master 127.24.33.126:55767 requested a full tablet report, sending... I0108 18:39:20.675941 24747 ts_manager.cc:192] Registered new tserver with Master: c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 18:39:20.676712 24747 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.24.33.65:56193 WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 18:39:20.684481 24916 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 18:39:20.684587 24916 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 18:39:20.685397 24916 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 18:39:20.685423 24916 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.33.66 I0108 18:39:20.686439 24916 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.24.33.84:49727 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.24.33.66:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/info.pb --webserver_interface=127.24.33.66 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.24.33.126:55767 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24916 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.24.33.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 4dca6b02a5e7b2e589646e79bc6e509f27c13c3d build type RELEASE built by jenkins at 08 Jan 2020 18:26:25 PST on quasar-luegrb-1.vpc.cloudera.com I0108 18:39:20.688526 24916 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 18:39:20.689671 24916 hybrid_clock.cc:186] HybridClock initialized: now 1578537560689660 us; error 16 us; skew 500 ppm I0108 18:39:20.689741 24916 fs_manager.cc:263] Metadata directory not provided I0108 18:39:20.689761 24916 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 18:39:20.689802 24916 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 18:39:20.689810 24916 server_base.cc:453] Attempting to create new FS layout instead I0108 18:39:20.690472 24916 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/instance: uuid: "53135810f2d84759ac110009de750c3d" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.690589 24916 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/instance: uuid: "53135810f2d84759ac110009de750c3d" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.691447 24916 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/data/block_manager_instance: No such file or directory (error 2) I0108 18:39:20.693974 24916 fs_manager.cc:518] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s I0108 18:39:20.694119 24916 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 18:39:20.694186 24916 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 18:39:20.695148 24916 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0108 18:39:20.695174 24916 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal uuid: "53135810f2d84759ac110009de750c3d" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.695204 24916 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 18:39:20.711840 24916 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 18:39:20.712059 24916 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0108 18:39:20.712071 24916 ts_tablet_manager.cc:415] Registered 0 tablets I0108 18:39:20.717552 24916 rpc_server.cc:211] RPC server started. Bound to: 127.24.33.66:33490 I0108 18:39:20.717788 24916 webserver.cc:382] Webserver started at http://127.24.33.66:50336/ using document root and password file I0108 18:39:20.717937 24916 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/info.pb I0108 18:39:20.720546 25037 heartbeater.cc:322] Connected to a master server at 127.24.33.126:55767 I0108 18:39:20.720566 25037 heartbeater.cc:406] Registering TS with master... I0108 18:39:20.720594 25037 heartbeater.cc:455] Master 127.24.33.126:55767 requested a full tablet report, sending... I0108 18:39:20.720729 24747 ts_manager.cc:192] Registered new tserver with Master: 53135810f2d84759ac110009de750c3d (127.24.33.66:33490) I0108 18:39:20.721179 24747 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.24.33.66:34965 I0108 18:39:20.725370 24709 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24916 I0108 18:39:20.725592 24709 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.24.33.67:0 --local_ip_for_outbound_sockets=127.24.33.67 --webserver_interface=127.24.33.67 --webserver_port=0 --tserver_master_addrs=127.24.33.126:55767 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.24.33.84:49727 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 18:39:20.736091 25040 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 18:39:20.736209 25040 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 18:39:20.737015 25040 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 18:39:20.737046 25040 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.33.67 I0108 18:39:20.738113 25040 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.24.33.84:49727 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.24.33.67:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/info.pb --webserver_interface=127.24.33.67 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.24.33.126:55767 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.25040 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.24.33.67 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 4dca6b02a5e7b2e589646e79bc6e509f27c13c3d build type RELEASE built by jenkins at 08 Jan 2020 18:26:25 PST on quasar-luegrb-1.vpc.cloudera.com I0108 18:39:20.740216 25040 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 18:39:20.741312 25040 hybrid_clock.cc:186] HybridClock initialized: now 1578537560741300 us; error 16 us; skew 500 ppm I0108 18:39:20.741403 25040 fs_manager.cc:263] Metadata directory not provided I0108 18:39:20.741427 25040 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 18:39:20.741487 25040 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 18:39:20.741499 25040 server_base.cc:453] Attempting to create new FS layout instead I0108 18:39:20.742179 25040 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/instance: uuid: "61b86269d42a403f94c0a049429fa247" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.742341 25040 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/instance: uuid: "61b86269d42a403f94c0a049429fa247" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.743153 25040 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/data/block_manager_instance: No such file or directory (error 2) I0108 18:39:20.745950 25040 fs_manager.cc:518] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s I0108 18:39:20.746090 25040 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 18:39:20.746158 25040 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 18:39:20.747107 25040 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0108 18:39:20.747133 25040 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal uuid: "61b86269d42a403f94c0a049429fa247" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:39:20.747164 25040 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 18:39:20.765580 25040 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 18:39:20.765787 25040 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0108 18:39:20.765802 25040 ts_tablet_manager.cc:415] Registered 0 tablets I0108 18:39:20.771139 25040 rpc_server.cc:211] RPC server started. Bound to: 127.24.33.67:35103 I0108 18:39:20.771383 25040 webserver.cc:382] Webserver started at http://127.24.33.67:37527/ using document root and password file I0108 18:39:20.771518 25040 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/data/info.pb I0108 18:39:20.774101 25161 heartbeater.cc:322] Connected to a master server at 127.24.33.126:55767 I0108 18:39:20.774121 25161 heartbeater.cc:406] Registering TS with master... I0108 18:39:20.774158 25161 heartbeater.cc:455] Master 127.24.33.126:55767 requested a full tablet report, sending... I0108 18:39:20.774294 24747 ts_manager.cc:192] Registered new tserver with Master: 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:39:20.774786 24747 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.24.33.67:59846 I0108 18:39:20.777359 24709 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 25040 I0108 18:39:20.780412 24709 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0108 18:39:20.785240 24747 catalog_manager.cc:1454] Servicing CreateTable request from {username='jenkins'} at 127.24.33.126:43341: name: "TestTable" schema { columns { name: "rand_key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "link_to" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "insert_ts" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "updated" type: BOOL is_key: false is_nullable: false read_default_value: "\000" encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU" } partition_schema { range_schema { columns { name: "rand_key" } } } W0108 18:39:20.785308 24747 catalog_manager.cc:1619] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0108 18:39:20.789290 25098 tablet_service.cc:1046] Processing CreateTablet for tablet 50603d14716f4c4bb514a91e103fbf64 (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0108 18:39:20.789301 25097 tablet_service.cc:1046] Processing CreateTablet for tablet fdf8e522824640c3ab9385967c93743a (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0108 18:39:20.789300 25096 tablet_service.cc:1046] Processing CreateTablet for tablet dbe8c3d61048482fb282021487b845a6 (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0108 18:39:20.789381 25097 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet fdf8e522824640c3ab9385967c93743a. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.789492 24972 tablet_service.cc:1046] Processing CreateTablet for tablet dbe8c3d61048482fb282021487b845a6 (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0108 18:39:20.789499 24974 tablet_service.cc:1046] Processing CreateTablet for tablet 50603d14716f4c4bb514a91e103fbf64 (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0108 18:39:20.789561 24973 tablet_service.cc:1046] Processing CreateTablet for tablet fdf8e522824640c3ab9385967c93743a (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0108 18:39:20.789585 24972 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet dbe8c3d61048482fb282021487b845a6. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.789619 24974 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 50603d14716f4c4bb514a91e103fbf64. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.789681 25096 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet dbe8c3d61048482fb282021487b845a6. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.791074 24849 tablet_service.cc:1046] Processing CreateTablet for tablet 50603d14716f4c4bb514a91e103fbf64 (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0108 18:39:20.791141 24849 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 50603d14716f4c4bb514a91e103fbf64. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.791327 25176 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Bootstrap starting. I0108 18:39:20.791544 25098 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 50603d14716f4c4bb514a91e103fbf64. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.791610 25176 tablet_bootstrap.cc:590] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: No blocks or log segments found. Creating new log. I0108 18:39:20.791751 25176 log.cc:815] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Log is configured to *not* fsync() on all Append() calls I0108 18:39:20.791834 24973 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet fdf8e522824640c3ab9385967c93743a. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.791893 25177 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Bootstrap starting. I0108 18:39:20.792155 25177 tablet_bootstrap.cc:590] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: No blocks or log segments found. Creating new log. I0108 18:39:20.792315 25177 log.cc:815] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Log is configured to *not* fsync() on all Append() calls I0108 18:39:20.792954 25180 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Bootstrap starting. I0108 18:39:20.792979 24848 tablet_service.cc:1046] Processing CreateTablet for tablet fdf8e522824640c3ab9385967c93743a (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0108 18:39:20.793015 24848 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet fdf8e522824640c3ab9385967c93743a. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.793179 25180 tablet_bootstrap.cc:590] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: No blocks or log segments found. Creating new log. I0108 18:39:20.793309 25180 log.cc:815] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Log is configured to *not* fsync() on all Append() calls I0108 18:39:20.793557 24847 tablet_service.cc:1046] Processing CreateTablet for tablet dbe8c3d61048482fb282021487b845a6 (table=TestTable [id=1354201f0f5d47fea15e1a86fd423353]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0108 18:39:20.793606 24847 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet dbe8c3d61048482fb282021487b845a6. 1 dirs total, 0 dirs full, 0 dirs failed I0108 18:39:20.798560 25176 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: No bootstrap required, opened a new log I0108 18:39:20.798593 25176 ts_tablet_manager.cc:1105] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s I0108 18:39:20.798857 25180 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: No bootstrap required, opened a new log I0108 18:39:20.798879 25180 ts_tablet_manager.cc:1105] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.000s I0108 18:39:20.799037 25177 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: No bootstrap required, opened a new log I0108 18:39:20.799064 25177 ts_tablet_manager.cc:1105] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.000s I0108 18:39:20.799991 25180 raft_consensus.cc:341] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.800006 25180 raft_consensus.cc:367] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.800002 25176 raft_consensus.cc:341] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.800012 25180 raft_consensus.cc:712] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ad452ff3b64d949298a50527e1fbb0, State: Initialized, Role: FOLLOWER I0108 18:39:20.800025 25176 raft_consensus.cc:367] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.800035 25176 raft_consensus.cc:712] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b86269d42a403f94c0a049429fa247, State: Initialized, Role: FOLLOWER I0108 18:39:20.800079 25180 consensus_queue.cc:229] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.800094 25176 consensus_queue.cc:229] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.800180 24912 heartbeater.cc:447] Master 127.24.33.126:55767 was elected leader, sending a full tablet report... I0108 18:39:20.800186 25180 ts_tablet_manager.cc:1133] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0108 18:39:20.800190 25176 ts_tablet_manager.cc:1133] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0108 18:39:20.800205 25180 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Bootstrap starting. I0108 18:39:20.800212 25176 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Bootstrap starting. I0108 18:39:20.800472 25176 tablet_bootstrap.cc:590] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: No blocks or log segments found. Creating new log. I0108 18:39:20.800529 25177 raft_consensus.cc:341] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.800532 25180 tablet_bootstrap.cc:590] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: No blocks or log segments found. Creating new log. I0108 18:39:20.800541 25177 raft_consensus.cc:367] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.800556 25177 raft_consensus.cc:712] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53135810f2d84759ac110009de750c3d, State: Initialized, Role: FOLLOWER I0108 18:39:20.800609 25177 consensus_queue.cc:229] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.800660 25161 heartbeater.cc:447] Master 127.24.33.126:55767 was elected leader, sending a full tablet report... I0108 18:39:20.800689 25177 ts_tablet_manager.cc:1133] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0108 18:39:20.800710 25177 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Bootstrap starting. I0108 18:39:20.800827 25037 heartbeater.cc:447] Master 127.24.33.126:55767 was elected leader, sending a full tablet report... I0108 18:39:20.801002 25177 tablet_bootstrap.cc:590] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: No blocks or log segments found. Creating new log. I0108 18:39:20.801299 25176 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: No bootstrap required, opened a new log I0108 18:39:20.801323 25176 ts_tablet_manager.cc:1105] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0108 18:39:20.801389 25180 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: No bootstrap required, opened a new log I0108 18:39:20.801411 25176 raft_consensus.cc:341] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.801419 25176 raft_consensus.cc:367] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.801422 25176 raft_consensus.cc:712] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b86269d42a403f94c0a049429fa247, State: Initialized, Role: FOLLOWER I0108 18:39:20.801421 25180 ts_tablet_manager.cc:1105] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0108 18:39:20.801463 25176 consensus_queue.cc:229] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.801501 25176 ts_tablet_manager.cc:1133] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:39:20.801524 25176 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Bootstrap starting. I0108 18:39:20.801539 25180 raft_consensus.cc:341] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.801550 25180 raft_consensus.cc:367] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.801558 25180 raft_consensus.cc:712] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ad452ff3b64d949298a50527e1fbb0, State: Initialized, Role: FOLLOWER I0108 18:39:20.801630 25180 consensus_queue.cc:229] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.801672 25180 ts_tablet_manager.cc:1133] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:39:20.801692 25180 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Bootstrap starting. I0108 18:39:20.801693 25176 tablet_bootstrap.cc:590] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: No blocks or log segments found. Creating new log. I0108 18:39:20.801970 25180 tablet_bootstrap.cc:590] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: No blocks or log segments found. Creating new log. I0108 18:39:20.802248 25177 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: No bootstrap required, opened a new log I0108 18:39:20.802281 25177 ts_tablet_manager.cc:1105] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0108 18:39:20.802376 25177 raft_consensus.cc:341] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.802392 25177 raft_consensus.cc:367] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.802398 25177 raft_consensus.cc:712] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53135810f2d84759ac110009de750c3d, State: Initialized, Role: FOLLOWER I0108 18:39:20.802435 25177 consensus_queue.cc:229] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.802495 25177 ts_tablet_manager.cc:1133] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:39:20.802525 25177 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Bootstrap starting. I0108 18:39:20.802708 25177 tablet_bootstrap.cc:590] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: No blocks or log segments found. Creating new log. I0108 18:39:20.803032 25176 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: No bootstrap required, opened a new log I0108 18:39:20.803061 25176 ts_tablet_manager.cc:1105] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I0108 18:39:20.803123 25180 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: No bootstrap required, opened a new log I0108 18:39:20.803138 25176 raft_consensus.cc:341] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.803143 25176 raft_consensus.cc:367] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.803148 25176 raft_consensus.cc:712] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b86269d42a403f94c0a049429fa247, State: Initialized, Role: FOLLOWER I0108 18:39:20.803148 25180 ts_tablet_manager.cc:1105] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I0108 18:39:20.803196 25176 consensus_queue.cc:229] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.803246 25180 raft_consensus.cc:341] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.803247 25176 ts_tablet_manager.cc:1133] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:39:20.803287 25180 raft_consensus.cc:367] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.803295 25180 raft_consensus.cc:712] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ad452ff3b64d949298a50527e1fbb0, State: Initialized, Role: FOLLOWER I0108 18:39:20.803314 25177 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: No bootstrap required, opened a new log I0108 18:39:20.803330 25177 ts_tablet_manager.cc:1105] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0108 18:39:20.803334 25180 consensus_queue.cc:229] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.803455 25177 raft_consensus.cc:341] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.803468 25177 raft_consensus.cc:367] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 18:39:20.803475 25177 raft_consensus.cc:712] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53135810f2d84759ac110009de750c3d, State: Initialized, Role: FOLLOWER I0108 18:39:20.803519 25177 consensus_queue.cc:229] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.803588 25180 ts_tablet_manager.cc:1133] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:39:20.803591 25177 ts_tablet_manager.cc:1133] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:39:20.865278 25185 raft_consensus.cc:477] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 18:39:20.865355 25185 raft_consensus.cc:499] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.865480 25185 leader_election.cc:293] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103), c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 18:39:20.867458 25118 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" is_pre_election: true I0108 18:39:20.867497 25118 raft_consensus.cc:2396] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 0. I0108 18:39:20.867573 24869 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" is_pre_election: true I0108 18:39:20.867600 24924 leader_election.cc:307] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: I0108 18:39:20.867612 24869 raft_consensus.cc:2396] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 0. I0108 18:39:20.867650 25185 raft_consensus.cc:2696] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0108 18:39:20.867669 25185 raft_consensus.cc:477] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0108 18:39:20.867676 25185 raft_consensus.cc:2949] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.868165 25185 raft_consensus.cc:499] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.868206 25185 leader_election.cc:293] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 election: Requested vote from peers 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103), c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 18:39:20.868309 25118 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" I0108 18:39:20.868328 25118 raft_consensus.cc:2949] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.868335 24869 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" I0108 18:39:20.868350 24869 raft_consensus.cc:2949] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.869223 25118 raft_consensus.cc:2396] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 1. I0108 18:39:20.869303 24924 leader_election.cc:307] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: I0108 18:39:20.869308 24869 raft_consensus.cc:2396] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 1. I0108 18:39:20.869359 25185 raft_consensus.cc:2696] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Leader election won for term 1 I0108 18:39:20.869442 25185 raft_consensus.cc:672] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 LEADER]: Becoming Leader. State: Replica: 53135810f2d84759ac110009de750c3d, State: Running, Role: LEADER I0108 18:39:20.869494 25185 consensus_queue.cc:206] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:39:20.869885 24746 catalog_manager.cc:4143] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d reported cstate change: term changed from 0 to 1, leader changed from to 53135810f2d84759ac110009de750c3d (127.24.33.66). New cstate: current_term: 1 leader_uuid: "53135810f2d84759ac110009de750c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } health_report { overall_health: UNKNOWN } } } W0108 18:39:20.922679 24913 tablet.cc:1792] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0108 18:39:20.933063 25182 raft_consensus.cc:477] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 18:39:20.933135 25182 raft_consensus.cc:499] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.933280 25182 leader_election.cc:293] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934), 53135810f2d84759ac110009de750c3d (127.24.33.66:33490) I0108 18:39:20.935468 24869 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "dbe8c3d61048482fb282021487b845a6" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" is_pre_election: true I0108 18:39:20.935473 24994 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "dbe8c3d61048482fb282021487b845a6" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true I0108 18:39:20.935500 24869 raft_consensus.cc:2396] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b86269d42a403f94c0a049429fa247 in term 0. I0108 18:39:20.935501 24994 raft_consensus.cc:2396] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b86269d42a403f94c0a049429fa247 in term 0. I0108 18:39:20.935606 25050 leader_election.cc:307] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: I0108 18:39:20.935652 25182 raft_consensus.cc:2696] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0108 18:39:20.935669 25182 raft_consensus.cc:477] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0108 18:39:20.935678 25182 raft_consensus.cc:2949] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.936727 25182 raft_consensus.cc:499] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.936771 25182 leader_election.cc:293] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 1 election: Requested vote from peers c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934), 53135810f2d84759ac110009de750c3d (127.24.33.66:33490) I0108 18:39:20.936873 24994 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "dbe8c3d61048482fb282021487b845a6" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" I0108 18:39:20.936888 24994 raft_consensus.cc:2949] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.936904 24869 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "dbe8c3d61048482fb282021487b845a6" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" I0108 18:39:20.936916 24869 raft_consensus.cc:2949] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.937587 24994 raft_consensus.cc:2396] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b86269d42a403f94c0a049429fa247 in term 1. I0108 18:39:20.937608 24869 raft_consensus.cc:2396] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b86269d42a403f94c0a049429fa247 in term 1. I0108 18:39:20.937675 25050 leader_election.cc:307] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: I0108 18:39:20.937722 25182 raft_consensus.cc:2696] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader election won for term 1 I0108 18:39:20.937808 25182 raft_consensus.cc:672] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [term 1 LEADER]: Becoming Leader. State: Replica: 61b86269d42a403f94c0a049429fa247, State: Running, Role: LEADER I0108 18:39:20.937863 25182 consensus_queue.cc:206] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.938295 24746 catalog_manager.cc:4143] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 reported cstate change: term changed from 0 to 1, leader changed from to 61b86269d42a403f94c0a049429fa247 (127.24.33.67). New cstate: current_term: 1 leader_uuid: "61b86269d42a403f94c0a049429fa247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } health_report { overall_health: HEALTHY } } } W0108 18:39:20.968295 25038 tablet.cc:1792] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0108 18:39:20.976402 25183 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 18:39:20.976485 25183 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.976635 25183 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:39:20.976869 25185 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 18:39:20.976930 25185 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.977102 24869 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" is_pre_election: true I0108 18:39:20.977118 25118 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" is_pre_election: true I0108 18:39:20.977128 24869 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 0. I0108 18:39:20.977140 25118 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 0. I0108 18:39:20.977214 24924 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: I0108 18:39:20.977293 25192 raft_consensus.cc:2696] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0108 18:39:20.977305 25192 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0108 18:39:20.977313 25192 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.977347 25185 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:39:20.978147 25192 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.978190 25192 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 election: Requested vote from peers c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:39:20.978389 25118 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" I0108 18:39:20.978391 24869 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" I0108 18:39:20.978404 25118 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.978404 24869 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 0 FOLLOWER]: Advancing to term 1 I0108 18:39:20.979110 25118 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 1. I0108 18:39:20.979133 24869 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 1. I0108 18:39:20.979255 24925 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, c6ad452ff3b64d949298a50527e1fbb0; no voters: I0108 18:39:20.979288 25118 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" is_pre_election: true I0108 18:39:20.979305 25118 raft_consensus.cc:2321] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6ad452ff3b64d949298a50527e1fbb0 in current term 1: Already voted for candidate 53135810f2d84759ac110009de750c3d in this term. I0108 18:39:20.979319 25192 raft_consensus.cc:2696] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Leader election won for term 1 I0108 18:39:20.979333 25192 raft_consensus.cc:672] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 1 LEADER]: Becoming Leader. State: Replica: 53135810f2d84759ac110009de750c3d, State: Running, Role: LEADER I0108 18:39:20.979387 25192 consensus_queue.cc:206] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:39:20.979802 24746 catalog_manager.cc:4143] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d reported cstate change: term changed from 0 to 1, leader changed from to 53135810f2d84759ac110009de750c3d (127.24.33.66). New cstate: current_term: 1 leader_uuid: "53135810f2d84759ac110009de750c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } health_report { overall_health: UNKNOWN } } } I0108 18:39:20.980218 24994 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true I0108 18:39:20.980329 24800 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ad452ff3b64d949298a50527e1fbb0; no voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247 I0108 18:39:20.980373 25183 raft_consensus.cc:2644] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0108 18:39:21.011565 24709 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0108 18:39:21.011907 24709 ts_itest-base.cc:233] Waiting for 3 tablets on tserver c6ad452ff3b64d949298a50527e1fbb0 to finish bootstrapping I0108 18:39:21.013811 24709 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 53135810f2d84759ac110009de750c3d to finish bootstrapping I0108 18:39:21.015507 24709 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 61b86269d42a403f94c0a049429fa247 to finish bootstrapping I0108 18:39:21.017946 24709 linked_list-test-util.h:512] Running for: 900.000s I0108 18:39:21.017957 24709 linked_list-test-util.h:513] Sampling every 300000000 us I0108 18:39:21.018043 24709 linked_list-test-util.h:519] Written 0 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 0 Mean: 0 Percentiles: 0% (min) = 0 25% = 0 50% (med) = 0 75% = 0 95% = 0 99% = 0 99.9% = 0 99.99% = 0 100% (max) = 0 I0108 18:39:21.020081 25199 linked_list-test-util.h:379] Curl thread will poll the following URLs every 1000 ms: I0108 18:39:21.020097 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/maintenance-manager I0108 18:39:21.020102 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/threadz I0108 18:39:21.020105 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/metrics I0108 18:39:21.020117 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/tablet-servers I0108 18:39:21.020121 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/scans I0108 18:39:21.020124 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/mem-trackers I0108 18:39:21.020126 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/tablets I0108 18:39:21.020129 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/tablets I0108 18:39:21.020133 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/stacks I0108 18:39:21.020135 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/threadz?group=all I0108 18:39:21.020138 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/mem-trackers I0108 18:39:21.020141 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/stacks I0108 18:39:21.020144 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/scans I0108 18:39:21.020148 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/mem-trackers I0108 18:39:21.020150 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/transactions?tablet_id=dbe8c3d61048482fb282021487b845a6 I0108 18:39:21.020153 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/scans I0108 18:39:21.020156 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/tables I0108 18:39:21.020159 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/maintenance-manager I0108 18:39:21.020162 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/stacks I0108 18:39:21.020165 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/mem-trackers I0108 18:39:21.020169 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/dump-entities I0108 18:39:21.020172 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/transactions?tablet_id=dbe8c3d61048482fb282021487b845a6 I0108 18:39:21.020175 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/maintenance-manager I0108 18:39:21.020179 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/threadz I0108 18:39:21.020181 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/threadz?group=all I0108 18:39:21.020184 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/tablets I0108 18:39:21.020189 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/transactions?tablet_id=dbe8c3d61048482fb282021487b845a6 I0108 18:39:21.020191 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/masters I0108 18:39:21.020195 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/metrics I0108 18:39:21.020197 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/stacks I0108 18:39:21.020200 25199 linked_list-test-util.h:382] http://127.24.33.66:50336/threadz?group=all I0108 18:39:21.020203 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/threadz I0108 18:39:21.020207 25199 linked_list-test-util.h:382] http://127.24.33.67:37527/threadz?group=all I0108 18:39:21.020210 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/threadz I0108 18:39:21.020213 25199 linked_list-test-util.h:382] http://127.24.33.126:54254/metrics I0108 18:39:21.020216 25199 linked_list-test-util.h:382] http://127.24.33.65:35659/metrics W0108 18:39:21.021857 25162 tablet.cc:1792] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Can't schedule compaction. Clean time has not been advanced past its initial value. I0108 18:39:21.022150 24994 raft_consensus.cc:1195] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Refusing update from remote peer 61b86269d42a403f94c0a049429fa247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 18:39:21.022300 25195 consensus_queue.cc:982] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 18:39:21.022296 24869 raft_consensus.cc:1195] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Refusing update from remote peer 61b86269d42a403f94c0a049429fa247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 18:39:21.022462 25195 consensus_queue.cc:982] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 18:39:21.022775 25118 raft_consensus.cc:1195] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Refusing update from remote peer 53135810f2d84759ac110009de750c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 18:39:21.022927 25185 consensus_queue.cc:982] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 18:39:21.023453 24868 raft_consensus.cc:1195] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Refusing update from remote peer 53135810f2d84759ac110009de750c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 18:39:21.023614 25192 consensus_queue.cc:982] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 18:39:21.023680 25117 raft_consensus.cc:1195] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Refusing update from remote peer 53135810f2d84759ac110009de750c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 18:39:21.023779 25205 mvcc.cc:203] Tried to move safe_time back from 6465689849945518080 to 6465689849601646592. Current Snapshot: MvccSnapshot[committed={T|T < 6465689849945518080}] I0108 18:39:21.023897 25192 consensus_queue.cc:982] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 18:39:21.024339 24868 raft_consensus.cc:1195] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Refusing update from remote peer 53135810f2d84759ac110009de750c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 18:39:21.024513 25216 mvcc.cc:203] Tried to move safe_time back from 6465689849945518080 to 6465689849601646592. Current Snapshot: MvccSnapshot[committed={T|T < 6465689849945518080}] I0108 18:39:21.024543 25192 consensus_queue.cc:982] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 18:39:21.024703 25204 mvcc.cc:203] Tried to move safe_time back from 6465689849948143616 to 6465689849321627648. Current Snapshot: MvccSnapshot[committed={T|T < 6465689849948143616}] ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 10000 Mean: 1644.83 I0108 18:39:39.041826 24709 linked_list-test-util.h:519] Written 500000 rows in chain Percentiles: 0% (min) = 804 25% = 1313 50% (med) = 1455 75% = 1639 95% = 2023 99% = 6116 99.9% = 34912 99.99% = 46944 100% (max) = 52406 I0108 18:39:57.900951 24709 linked_list-test-util.h:519] Written 1000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 20000 Mean: 1683.72 Percentiles: 0% (min) = 804 25% = 1335 50% (med) = 1482 75% = 1673 95% = 2104 99% = 6128 99.9% = 38016 99.99% = 53376 100% (max) = 55895 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 30000 Mean: 1701.48 Percentiles: 0% (min) = 804 25% = 1351 50% (med) = 1506 75% = 1702 95% = 2130 99% = 5920 99.9% = 35872 99.99% = 53376 100% (max) = 55895 I0108 18:40:16.957940 24709 linked_list-test-util.h:519] Written 1500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 40000 Mean: 1713.19 Percentiles: 0% (min) = 804 25% = 1362 50% (med) = 1519 75% = 1717 95% = 2144 99% = 5908 99.9% = 36704 99.99% = 53376 100% (max) = 55895 I0108 18:40:36.085888 24709 linked_list-test-util.h:519] Written 2000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 50000 Mean: 1723.84 Percentiles: 0% (min) = 804 25% = 1370 50% (med) = 1528 75% = 1731 95% = 2166 99% = 5864 99.9% = 36256 99.99% = 53440 100% (max) = 55895 I0108 18:40:55.268193 24709 linked_list-test-util.h:519] Written 2500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 60000 Mean: 1732.58 Percentiles: 0% (min) = 804 25% = 1374 50% (med) = 1532 75% = 1737 95% = 2182 99% = 5940 99.9% = 37024 99.99% = 53472 100% (max) = 61458 I0108 18:41:14.631893 24709 linked_list-test-util.h:519] Written 3000000 rows in chain I0108 18:41:20.634428 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360 I0108 18:41:20.654963 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.001s sys 0.003s Metrics: {"bytes_written":7592,"cfile_init":1,"dirs.queue_time_us":5282,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":494,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":166,"lbm_writes_lt_1ms":27,"rows_written":6,"thread_start_us":54,"threads_started":2,"wal-append.queue_time_us":8793} I0108 18:41:20.655174 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 591 bytes on disk I0108 18:41:20.655287 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 18:41:20.814651 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033337 I0108 18:41:20.876212 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033354 I0108 18:41:21.013038 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033392 I0108 18:41:24.196902 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 3.321s user 0.870s sys 0.042s Metrics: {"bytes_written":9339438,"cfile_init":1,"compiler_manager_pool.queue_time_us":7190,"compiler_manager_pool.run_cpu_time_us":172762,"compiler_manager_pool.run_wall_time_us":496403,"delete_count":0,"dirs.queue_time_us":6922,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":37540,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":106024,"lbm_writes_1-10_ms":23,"lbm_writes_lt_1ms":928,"reinsert_count":0,"rows_written":526961,"thread_start_us":71,"threads_started":2,"update_count":1454} I0108 18:41:24.197080 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 33562270 bytes of WAL I0108 18:41:24.197155 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 4 log segments from log reader I0108 18:41:24.197194 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000001 (ops 1-12488) I0108 18:41:24.197280 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000002 (ops 12489-24882) I0108 18:41:24.197345 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000003 (ops 24883-37265) I0108 18:41:24.197402 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000004 (ops 37266-49642) I0108 18:41:24.246662 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.049s user 0.003s sys 0.008s Metrics: {} I0108 18:41:24.246811 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 10455615 bytes on disk I0108 18:41:24.249819 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:41:24.250862 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.034291 I0108 18:41:27.533125 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 3.282s user 0.831s sys 0.086s Metrics: {"bytes_written":9577873,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":8530,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":26606,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":96457,"lbm_writes_1-10_ms":25,"lbm_writes_lt_1ms":948,"reinsert_count":0,"rows_written":540176,"thread_start_us":31,"threads_started":1,"update_count":2541} I0108 18:41:27.533327 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41949352 bytes of WAL I0108 18:41:27.533404 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:41:27.533457 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000001 (ops 1-12531) I0108 18:41:27.533571 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000002 (ops 12532-24948) I0108 18:41:27.537031 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000003 (ops 24949-37367) I0108 18:41:27.537541 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000004 (ops 37368-49813) I0108 18:41:27.537621 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000005 (ops 49814-62266) I0108 18:41:27.659098 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.126s user 0.000s sys 0.012s Metrics: {} I0108 18:41:27.659315 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.035238 I0108 18:41:34.654914 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 13.840s user 3.591s sys 0.276s Metrics: {"bytes_written":37302249,"cfile_init":2,"compiler_manager_pool.queue_time_us":221,"compiler_manager_pool.run_cpu_time_us":171346,"compiler_manager_pool.run_wall_time_us":577214,"delete_count":0,"dirs.queue_time_us":9288,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":109271,"drs_written":2,"lbm_read_time_us":37,"lbm_reads_lt_1ms":8,"lbm_write_time_us":371971,"lbm_writes_1-10_ms":73,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3681,"reinsert_count":0,"rows_written":2106129,"thread_start_us":117,"threads_started":3,"update_count":4813} I0108 18:41:34.655112 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 83675789 bytes of WAL I0108 18:41:34.655217 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:41:34.655285 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000001 (ops 1-5981) I0108 18:41:34.655398 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000002 (ops 5982-11810) I0108 18:41:34.655486 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000003 (ops 11811-17726) I0108 18:41:34.655767 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000004 (ops 17727-23629) I0108 18:41:34.655881 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000005 (ops 23630-29474) I0108 18:41:34.655944 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000006 (ops 29475-35311) I0108 18:41:34.656535 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000007 (ops 35312-41162) I0108 18:41:34.656635 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000008 (ops 41163-47089) I0108 18:41:34.658565 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000009 (ops 47090-53025) I0108 18:41:34.659580 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000010 (ops 53026-58941) I0108 18:41:34.758683 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.103s user 0.004s sys 0.019s Metrics: {} I0108 18:41:34.758831 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 41883289 bytes on disk I0108 18:41:34.758997 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":23,"lbm_reads_lt_1ms":8} I0108 18:41:34.760285 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.037210 W0108 18:41:35.106406 24807 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.244s user 0.000s sys 0.000s I0108 18:41:35.107417 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 14.094s user 3.570s sys 0.234s Metrics: {"bytes_written":37381569,"cfile_init":2,"compiler_manager_pool.queue_time_us":9680,"compiler_manager_pool.run_cpu_time_us":166425,"compiler_manager_pool.run_wall_time_us":552417,"delete_count":0,"dirs.queue_time_us":5599,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":212203,"drs_written":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8,"lbm_write_time_us":502682,"lbm_writes_1-10_ms":85,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3673,"reinsert_count":0,"rows_written":2108978,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":3,"update_count":7662} I0108 18:41:35.107632 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.037307 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 70000 Mean: 1766.07 Percentiles: 0% (min) = 804 25% = 1383 50% (med) = 1544 75% = 1756 95% = 2292 99% = 6540 99.9% = 38816 99.99% = 61376 100% (max) = 72287 I0108 18:41:35.618144 24709 linked_list-test-util.h:519] Written 3500000 rows in chain I0108 18:41:38.476220 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 3.716s user 0.890s sys 0.095s Metrics: {"bytes_written":10237659,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12049,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":32820,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":124420,"lbm_writes_1-10_ms":32,"lbm_writes_lt_1ms":1010,"reinsert_count":0,"rows_written":579540,"thread_start_us":32,"threads_started":1,"update_count":194} I0108 18:41:38.476428 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41953914 bytes of WAL I0108 18:41:38.476503 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:41:38.476550 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000001 (ops 1-12499) I0108 18:41:38.476629 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000002 (ops 12500-24903) I0108 18:41:38.476706 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000003 (ops 24904-37294) I0108 18:41:38.476766 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000004 (ops 37295-49682) I0108 18:41:38.476830 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000005 (ops 49683-62088) I0108 18:41:38.505796 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.029s user 0.000s sys 0.012s Metrics: {} I0108 18:41:38.506688 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 11547296 bytes on disk I0108 18:41:38.506817 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:41:38.506906 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.038251 I0108 18:41:38.771695 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 3.664s user 0.875s sys 0.097s Metrics: {"bytes_written":10291378,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":24370,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":18454,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":122105,"lbm_writes_1-10_ms":32,"lbm_writes_lt_1ms":1013,"reinsert_count":0,"rows_written":581486,"thread_start_us":31,"threads_started":1,"update_count":1583} I0108 18:41:38.771878 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 84076765 bytes of WAL I0108 18:41:38.771958 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:41:38.772044 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000001 (ops 1-5995) I0108 18:41:38.772126 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000002 (ops 5996-11858) I0108 18:41:38.772234 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000003 (ops 11859-17800) I0108 18:41:38.772297 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000004 (ops 17801-23682) I0108 18:41:38.772437 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000005 (ops 23683-29617) I0108 18:41:38.772497 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000006 (ops 29618-35554) I0108 18:41:38.772549 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000007 (ops 35555-41465) I0108 18:41:38.772636 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000008 (ops 41466-47389) I0108 18:41:38.772689 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000009 (ops 47390-53328) I0108 18:41:38.772737 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000010 (ops 53329-59253) I0108 18:41:38.880789 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.109s user 0.000s sys 0.024s Metrics: {} I0108 18:41:38.880959 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41954947 bytes of WAL I0108 18:41:38.881033 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:41:38.883091 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000001 (ops 1-12507) I0108 18:41:38.883180 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000002 (ops 12508-24902) I0108 18:41:38.885187 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000003 (ops 24903-37312) I0108 18:41:38.886044 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000004 (ops 37313-49739) I0108 18:41:38.886199 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000005 (ops 49740-62173) I0108 18:41:38.926864 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.046s user 0.000s sys 0.014s Metrics: {} I0108 18:41:38.927031 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.038368 I0108 18:41:41.587884 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 13.928s user 3.705s sys 0.271s Metrics: {"bytes_written":39221478,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":10428,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":70042,"drs_written":2,"lbm_read_time_us":25,"lbm_reads_lt_1ms":8,"lbm_write_time_us":403298,"lbm_writes_1-10_ms":96,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3838,"reinsert_count":0,"rows_written":2210215,"thread_start_us":64,"threads_started":2,"update_count":14418} I0108 18:41:41.589747 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 92425334 bytes of WAL I0108 18:41:41.589826 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 11 log segments from log reader I0108 18:41:41.589881 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000001 (ops 1-6005) I0108 18:41:41.589995 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000002 (ops 6006-11881) I0108 18:41:41.592005 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000003 (ops 11882-17819) I0108 18:41:41.592157 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000004 (ops 17820-23683) I0108 18:41:41.592269 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000005 (ops 23684-29616) I0108 18:41:41.592360 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000006 (ops 29617-35555) I0108 18:41:41.592435 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000007 (ops 35556-41481) I0108 18:41:41.592572 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000008 (ops 41482-47397) I0108 18:41:41.592881 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000009 (ops 47398-53352) I0108 18:41:41.592955 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000010 (ops 53353-59291) I0108 18:41:41.593014 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000011 (ops 59292-65171) I0108 18:41:41.645941 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.056s user 0.001s sys 0.025s Metrics: {} I0108 18:41:41.664332 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 43959367 bytes on disk I0108 18:41:41.664548 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":34,"lbm_reads_lt_1ms":8} I0108 18:41:41.664644 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 10722558 bytes on disk I0108 18:41:41.664785 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:41:41.735893 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 3.229s user 0.925s sys 0.048s Metrics: {"bytes_written":10543445,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":8071,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":38456,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":150234,"lbm_writes_1-10_ms":27,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":1041,"reinsert_count":0,"rows_written":595359,"thread_start_us":37,"threads_started":1,"update_count":2622} I0108 18:41:41.736069 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41952115 bytes of WAL I0108 18:41:41.736143 24932 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:41:41.736191 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000001 (ops 1-12481) I0108 18:41:41.736310 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000002 (ops 12482-24850) I0108 18:41:41.736387 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000003 (ops 24851-37221) I0108 18:41:41.736465 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000004 (ops 37222-49622) I0108 18:41:41.736531 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000005 (ops 49623-62020) I0108 18:41:41.759968 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.024s user 0.000s sys 0.012s Metrics: {} I0108 18:41:41.762163 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 11855141 bytes on disk I0108 18:41:41.762321 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 18:41:42.144623 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 3.217s user 0.909s sys 0.069s Metrics: {"bytes_written":10526820,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":3254,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":35101,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":110754,"lbm_writes_1-10_ms":29,"lbm_writes_lt_1ms":1043,"reinsert_count":0,"rows_written":595923,"thread_start_us":31,"threads_started":1,"update_count":364} I0108 18:41:42.144798 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41954508 bytes of WAL I0108 18:41:42.144857 24807 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:41:42.144876 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000001 (ops 1-12499) I0108 18:41:42.144913 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000002 (ops 12500-24904) I0108 18:41:42.144981 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000003 (ops 24905-37313) I0108 18:41:42.146306 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000004 (ops 37314-49700) I0108 18:41:42.146550 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000005 (ops 49701-62118) I0108 18:41:42.170976 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.026s user 0.002s sys 0.011s Metrics: {} I0108 18:41:42.171061 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 41918647 bytes on disk I0108 18:41:42.171459 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":203,"lbm_reads_lt_1ms":8} I0108 18:41:42.171572 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 11875883 bytes on disk I0108 18:41:42.172170 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0108 18:41:42.173357 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 11584489 bytes on disk I0108 18:41:42.173480 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":15,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 80000 Mean: 1786.29 Percentiles: 0% (min) = 804 25% = 1391 50% (med) = 1555 75% = 1772 95% = 2350 99% = 6736 99.9% = 40416 99.99% = 61440 100% (max) = 72287 I0108 18:41:56.297196 24709 linked_list-test-util.h:519] Written 4000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 90000 Mean: 1795.43 Percentiles: 0% (min) = 804 25% = 1398 50% (med) = 1564 75% = 1782 95% = 2352 99% = 6648 99.9% = 41344 99.99% = 61856 100% (max) = 72287 I0108 18:42:16.565479 24709 linked_list-test-util.h:519] Written 4500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 100000 Mean: 1804.02 Percentiles: 0% (min) = 804 25% = 1407 50% (med) = 1575 75% = 1796 95% = 2356 99% = 6624 99.9% = 41312 99.99% = 61440 100% (max) = 72287 I0108 18:42:37.040455 24709 linked_list-test-util.h:519] Written 5000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 110000 Mean: 1812.26 Percentiles: 0% (min) = 804 25% = 1415 50% (med) = 1586 75% = 1809 95% = 2364 99% = 6572 99.9% = 40672 99.99% = 61376 100% (max) = 72287 I0108 18:42:57.601047 24709 linked_list-test-util.h:519] Written 5500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 120000 Mean: 1819.6 Percentiles: 0% (min) = 804 25% = 1421 50% (med) = 1592 75% = 1817 95% = 2370 99% = 6648 99.9% = 40736 99.99% = 60256 100% (max) = 72287 I0108 18:43:18.122738 24709 linked_list-test-util.h:519] Written 6000000 rows in chain I0108 18:43:24.265100 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033352 I0108 18:43:25.790488 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 1.525s user 0.691s sys 0.056s Metrics: {"bytes_written":8767088,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9858,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":24754,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":42223,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":898,"reinsert_count":0,"rows_written":498331,"thread_start_us":581,"threads_started":1,"update_count":634} I0108 18:43:25.790676 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41926789 bytes of WAL I0108 18:43:25.790851 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:43:25.790901 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000005 (ops 49643-62037) I0108 18:43:25.790980 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000006 (ops 62038-74365) I0108 18:43:25.791050 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000007 (ops 74366-86758) I0108 18:43:25.791116 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000008 (ops 86759-99166) I0108 18:43:25.791182 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000009 (ops 99167-111515) I0108 18:43:25.810348 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.019s user 0.000s sys 0.012s Metrics: {} I0108 18:43:25.810497 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.004682 I0108 18:43:28.934468 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 3.124s user 1.305s sys 0.114s Metrics: {"bytes_written":17979889,"cfile_cache_hit":48,"cfile_cache_hit_bytes":5770497,"cfile_cache_miss":2384,"cfile_cache_miss_bytes":66202999,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":19085,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":23691,"drs_written":1,"lbm_read_time_us":58722,"lbm_reads_1-10_ms":8,"lbm_reads_lt_1ms":2416,"lbm_write_time_us":107558,"lbm_writes_1-10_ms":13,"lbm_writes_lt_1ms":1810,"num_input_rowsets":2,"rows_written":1025292,"spinlock_wait_cycles":2432,"thread_start_us":525,"threads_started":9} I0108 18:43:28.934751 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.033723 I0108 18:43:30.431823 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 1.497s user 0.673s sys 0.051s Metrics: {"bytes_written":8845175,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1096,"dirs.run_cpu_time_us":49,"dirs.run_wall_time_us":49,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":117361,"lbm_writes_1-10_ms":6,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":897,"reinsert_count":0,"rows_written":501284,"thread_start_us":31,"threads_started":1,"update_count":3022} I0108 18:43:30.432006 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33559483 bytes of WAL I0108 18:43:30.432180 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:43:30.432231 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000006 (ops 62267-74646) I0108 18:43:30.432333 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000007 (ops 74647-87111) I0108 18:43:30.432401 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000008 (ops 87112-99527) I0108 18:43:30.432615 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000009 (ops 99528-111931) I0108 18:43:30.444550 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.012s user 0.000s sys 0.011s Metrics: {} I0108 18:43:30.444720 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 20851233 bytes on disk I0108 18:43:30.454367 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.010s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9494,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":2} I0108 18:43:30.455243 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.004510 I0108 18:43:33.644878 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 3.190s user 1.298s sys 0.138s Metrics: {"bytes_written":18260666,"cfile_cache_hit":44,"cfile_cache_hit_bytes":5122430,"cfile_cache_miss":2424,"cfile_cache_miss_bytes":68060042,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":92,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":45846,"drs_written":1,"lbm_read_time_us":248139,"lbm_reads_1-10_ms":17,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":2442,"lbm_write_time_us":118574,"lbm_writes_1-10_ms":4,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":1845,"num_input_rowsets":2,"rows_written":1041460,"spinlock_wait_cycles":2048,"thread_start_us":433,"threads_started":9} I0108 18:43:33.645042 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 21213044 bytes on disk I0108 18:43:33.645159 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:43:34.861971 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033391 I0108 18:43:35.276969 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033380 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 130000 Mean: 1840.11 Percentiles: 0% (min) = 804 25% = 1428 50% (med) = 1603 75% = 1834 95% = 2412 99% = 6812 99.9% = 41696 99.99% = 61408 100% (max) = 86845 I0108 18:43:40.564376 24709 linked_list-test-util.h:519] Written 6500000 rows in chain I0108 18:43:41.651836 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033351 I0108 18:43:45.816023 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 10.954s user 3.441s sys 0.269s Metrics: {"bytes_written":37743777,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":9834,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":92651,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":439500,"lbm_writes_1-10_ms":70,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":3744,"reinsert_count":0,"rows_written":2143121,"thread_start_us":63,"threads_started":2,"update_count":10941} I0108 18:43:45.816495 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 42727859 bytes on disk I0108 18:43:45.816749 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0108 18:43:45.816865 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.010813 W0108 18:43:46.896391 24807 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.354s user 0.001s sys 0.000s I0108 18:43:46.909145 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 11.632s user 3.440s sys 0.244s Metrics: {"bytes_written":37820330,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":13344,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":85278,"drs_written":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8,"lbm_write_time_us":285134,"lbm_writes_1-10_ms":61,"lbm_writes_lt_1ms":3761,"reinsert_count":0,"rows_written":2144849,"thread_start_us":63,"threads_started":2,"update_count":15518} I0108 18:43:46.909351 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.010767 I0108 18:43:54.276018 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 12.624s user 3.506s sys 0.292s Metrics: {"bytes_written":37598214,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":16910,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":79237,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":379527,"lbm_writes_1-10_ms":71,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":3740,"reinsert_count":0,"rows_written":2139018,"spinlock_wait_cycles":250624,"thread_start_us":107,"threads_started":2,"update_count":4052} I0108 18:43:54.277835 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 92747251 bytes of WAL I0108 18:43:54.278051 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 11 log segments from log reader I0108 18:43:54.278115 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000012 (ops 65172-71156) I0108 18:43:54.278211 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000013 (ops 71157-77040) I0108 18:43:54.278714 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000014 (ops 77041-83053) I0108 18:43:54.278789 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000015 (ops 83054-89057) I0108 18:43:54.278867 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000016 (ops 89058-95013) I0108 18:43:54.279119 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000017 (ops 95014-100948) I0108 18:43:54.279249 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000018 (ops 100949-106852) I0108 18:43:54.279364 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000019 (ops 106853-112770) I0108 18:43:54.279431 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000020 (ops 112771-118700) I0108 18:43:54.279525 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000021 (ops 118701-124602) I0108 18:43:54.279631 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000022 (ops 124603-130554) I0108 18:43:54.378360 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.099s user 0.000s sys 0.026s Metrics: {} I0108 18:43:54.378520 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.009803 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 140000 Mean: 1867 Percentiles: 0% (min) = 804 25% = 1435 50% (med) = 1615 75% = 1855 95% = 2504 99% = 7084 99.9% = 42080 99.99% = 61856 100% (max) = 86845 I0108 18:44:03.979497 24709 linked_list-test-util.h:519] Written 7000000 rows in chain I0108 18:44:08.535763 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 22.719s user 5.774s sys 0.569s Metrics: {"bytes_written":74466556,"cfile_cache_hit":204,"cfile_cache_hit_bytes":27928651,"cfile_cache_miss":9836,"cfile_cache_miss_bytes":270886217,"cfile_init":19,"delta_iterators_relevant":16,"dirs.queue_time_us":35835,"dirs.run_cpu_time_us":326,"dirs.run_wall_time_us":449497,"drs_written":3,"lbm_read_time_us":709049,"lbm_reads_1-10_ms":100,"lbm_reads_10-100_ms":5,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":9806,"lbm_write_time_us":914489,"lbm_writes_1-10_ms":176,"lbm_writes_10-100_ms":3,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":7359,"num_input_rowsets":4,"rows_written":4249250,"spinlock_wait_cycles":91465472,"thread_start_us":431,"threads_started":11} I0108 18:44:08.535955 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 92481225 bytes of WAL I0108 18:44:08.536031 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 11 log segments from log reader I0108 18:44:08.536078 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000011 (ops 58942-64786) I0108 18:44:08.536152 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000012 (ops 64787-70681) I0108 18:44:08.536224 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000013 (ops 70682-76599) I0108 18:44:08.536319 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000014 (ops 76600-82551) I0108 18:44:08.536392 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000015 (ops 82552-88549) I0108 18:44:08.536460 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000016 (ops 88550-94440) I0108 18:44:08.536528 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000017 (ops 94441-100328) I0108 18:44:08.536595 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000018 (ops 100329-106232) I0108 18:44:08.536665 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000019 (ops 106233-112151) I0108 18:44:08.536752 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000020 (ops 112152-118034) I0108 18:44:08.536814 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000021 (ops 118035-123954) I0108 18:44:08.603915 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.068s user 0.001s sys 0.026s Metrics: {} I0108 18:44:08.619408 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.041706 I0108 18:44:08.905537 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 21.996s user 5.773s sys 0.542s Metrics: {"bytes_written":74546439,"cfile_cache_hit":205,"cfile_cache_hit_bytes":28117611,"cfile_cache_miss":9843,"cfile_cache_miss_bytes":271050831,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":27785,"dirs.run_cpu_time_us":320,"dirs.run_wall_time_us":140417,"drs_written":3,"lbm_read_time_us":393292,"lbm_reads_1-10_ms":94,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":9831,"lbm_write_time_us":838355,"lbm_writes_1-10_ms":186,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":7352,"num_input_rowsets":4,"rows_written":4253827,"spinlock_wait_cycles":2674688,"thread_start_us":108,"threads_started":3} I0108 18:44:08.905671 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 92367268 bytes of WAL I0108 18:44:08.905732 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 11 log segments from log reader I0108 18:44:08.905750 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000011 (ops 59254-65138) I0108 18:44:08.905791 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000012 (ops 65139-71136) I0108 18:44:08.905830 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000013 (ops 71137-77033) I0108 18:44:08.905870 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000014 (ops 77034-83043) I0108 18:44:08.905910 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000015 (ops 83044-88917) I0108 18:44:08.905972 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000016 (ops 88918-94805) I0108 18:44:08.906028 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000017 (ops 94806-100697) I0108 18:44:08.906078 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000018 (ops 100698-106606) I0108 18:44:08.906121 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000019 (ops 106607-112524) I0108 18:44:08.906198 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000020 (ops 112525-118331) I0108 18:44:08.906256 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000021 (ops 118332-124277) I0108 18:44:08.947823 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.042s user 0.002s sys 0.026s Metrics: {} I0108 18:44:08.947975 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 86858591 bytes on disk I0108 18:44:09.011883 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.064s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":63651,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":7} I0108 18:44:09.012038 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.041733 I0108 18:44:11.548959 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.929s user 0.903s sys 0.079s Metrics: {"bytes_written":10607149,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4744,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":82836,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":351702,"lbm_writes_1-10_ms":22,"lbm_writes_10-100_ms":15,"lbm_writes_lt_1ms":1047,"reinsert_count":0,"rows_written":603506,"thread_start_us":30,"threads_started":1,"update_count":2649} I0108 18:44:11.549154 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 50420109 bytes of WAL I0108 18:44:11.549773 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 6 log segments from log reader I0108 18:44:11.549794 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000006 (ops 62089-74461) I0108 18:44:11.549944 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000007 (ops 74462-86859) I0108 18:44:11.550014 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000008 (ops 86860-99274) I0108 18:44:11.550073 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000009 (ops 99275-111692) I0108 18:44:11.550134 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000010 (ops 111693-124136) I0108 18:44:11.550200 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000011 (ops 124137-136527) I0108 18:44:11.640560 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.091s user 0.000s sys 0.020s Metrics: {} I0108 18:44:11.640797 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.003079 I0108 18:44:12.033692 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 3.022s user 0.936s sys 0.058s Metrics: {"bytes_written":10531135,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":14866,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":30144,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":301287,"lbm_writes_1-10_ms":38,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":1037,"reinsert_count":0,"rows_written":600660,"thread_start_us":31,"threads_started":1,"update_count":37} I0108 18:44:12.034999 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 50461935 bytes of WAL I0108 18:44:12.035334 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 6 log segments from log reader I0108 18:44:12.035352 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000006 (ops 62174-74579) I0108 18:44:12.035432 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000007 (ops 74580-87021) I0108 18:44:12.035465 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000008 (ops 87022-99415) I0108 18:44:12.035531 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000009 (ops 99416-111787) I0108 18:44:12.035565 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000010 (ops 111788-124259) I0108 18:44:12.035588 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000011 (ops 124260-136828) I0108 18:44:12.100136 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.065s user 0.001s sys 0.015s Metrics: {} I0108 18:44:12.100317 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 12017016 bytes on disk I0108 18:44:12.103410 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 18:44:12.103518 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.003083 I0108 18:44:16.643429 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 22.262s user 5.795s sys 0.577s Metrics: {"bytes_written":76195371,"cfile_cache_hit":227,"cfile_cache_hit_bytes":31605796,"cfile_cache_miss":10043,"cfile_cache_miss_bytes":274261872,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":15624,"dirs.run_cpu_time_us":304,"dirs.run_wall_time_us":125656,"drs_written":3,"lbm_read_time_us":1598422,"lbm_reads_1-10_ms":138,"lbm_reads_10-100_ms":20,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":9967,"lbm_write_time_us":760517,"lbm_writes_1-10_ms":174,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":7537,"num_input_rowsets":4,"rows_written":4349233,"spinlock_wait_cycles":52768768,"thread_start_us":1056,"threads_started":11} I0108 18:44:16.643535 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 88872418 bytes on disk I0108 18:44:16.643808 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":17,"lbm_reads_lt_1ms":12} I0108 18:44:16.889104 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 5.248s user 1.610s sys 0.177s Metrics: {"bytes_written":20703081,"cfile_cache_hit":48,"cfile_cache_hit_bytes":5643579,"cfile_cache_miss":2752,"cfile_cache_miss_bytes":77535043,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":7194,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":37491,"drs_written":1,"lbm_read_time_us":359521,"lbm_reads_1-10_ms":37,"lbm_reads_10-100_ms":10,"lbm_reads_lt_1ms":2745,"lbm_write_time_us":159769,"lbm_writes_1-10_ms":30,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":2070,"num_input_rowsets":2,"rows_written":1183046,"spinlock_wait_cycles":2617600,"thread_start_us":1574,"threads_started":9} I0108 18:44:16.889308 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.043098 I0108 18:44:17.024704 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 4.917s user 1.543s sys 0.151s Metrics: {"bytes_written":20689541,"cfile_cache_hit":48,"cfile_cache_hit_bytes":5662610,"cfile_cache_miss":2750,"cfile_cache_miss_bytes":77472932,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":4641,"dirs.run_cpu_time_us":116,"dirs.run_wall_time_us":64401,"drs_written":1,"lbm_read_time_us":507464,"lbm_reads_1-10_ms":27,"lbm_reads_10-100_ms":12,"lbm_reads_lt_1ms":2747,"lbm_write_time_us":163740,"lbm_writes_1-10_ms":26,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2072,"num_input_rowsets":2,"rows_written":1182146,"spinlock_wait_cycles":2432,"thread_start_us":535,"threads_started":8} I0108 18:44:17.027323 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.043023 I0108 18:44:19.508893 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.619s user 0.861s sys 0.062s Metrics: {"bytes_written":10760437,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":222,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":167363,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":259520,"lbm_writes_1-10_ms":7,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":1087,"reinsert_count":0,"rows_written":612277,"thread_start_us":31,"threads_started":1,"update_count":2447} I0108 18:44:19.509251 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 50347642 bytes of WAL I0108 18:44:19.509348 24932 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 6 log segments from log reader I0108 18:44:19.509402 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000006 (ops 62021-74422) I0108 18:44:19.509735 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000007 (ops 74423-86839) I0108 18:44:19.509807 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000008 (ops 86840-99280) I0108 18:44:19.509873 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000009 (ops 99281-111649) I0108 18:44:19.509934 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000010 (ops 111650-124016) I0108 18:44:19.510007 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000011 (ops 124017-136362) I0108 18:44:19.553853 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.044s user 0.002s sys 0.015s Metrics: {} I0108 18:44:19.554009 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 86768380 bytes on disk I0108 18:44:19.603313 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.576s user 0.885s sys 0.055s Metrics: {"bytes_written":10743620,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":23733,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":36869,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":224396,"lbm_writes_1-10_ms":9,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":1088,"reinsert_count":0,"rows_written":613019,"thread_start_us":31,"threads_started":1,"update_count":20} I0108 18:44:19.603420 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 50387762 bytes of WAL I0108 18:44:19.603497 24807 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 6 log segments from log reader I0108 18:44:19.603513 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000006 (ops 62119-74498) I0108 18:44:19.603575 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000007 (ops 74499-86903) I0108 18:44:19.603610 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000008 (ops 86904-99274) I0108 18:44:19.603631 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000009 (ops 99275-111699) I0108 18:44:19.603667 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000010 (ops 111700-124143) I0108 18:44:19.603699 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000011 (ops 124144-136545) I0108 18:44:19.613931 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.059s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":58710,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":6} I0108 18:44:19.614084 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 24231988 bytes on disk I0108 18:44:19.623997 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.010s user 0.000s sys 0.001s Metrics: {"cfile_init":1,"lbm_read_time_us":9532,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":2} I0108 18:44:19.624137 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 12237903 bytes on disk I0108 18:44:19.624308 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4} I0108 18:44:19.624406 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.002822 I0108 18:44:19.628995 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.025s user 0.000s sys 0.015s Metrics: {} I0108 18:44:19.629246 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 24218767 bytes on disk I0108 18:44:19.650386 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.021s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20066,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0108 18:44:19.650959 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.002818 I0108 18:44:21.019222 24709 linked_list-test-util.h:528] Sample at HT timestamp: 6465691078732034048 Inserted count: 7332500 I0108 18:44:25.152215 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 5.501s user 1.586s sys 0.160s Metrics: {"bytes_written":21150381,"cfile_cache_hit":52,"cfile_cache_hit_bytes":6112670,"cfile_cache_miss":2810,"cfile_cache_miss_bytes":78882704,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":250,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":54615,"drs_written":1,"lbm_read_time_us":504472,"lbm_reads_1-10_ms":34,"lbm_reads_10-100_ms":17,"lbm_reads_lt_1ms":2799,"lbm_write_time_us":150382,"lbm_writes_1-10_ms":27,"lbm_writes_lt_1ms":2117,"num_input_rowsets":2,"rows_written":1208942,"spinlock_wait_cycles":6390272,"thread_start_us":236,"threads_started":9} I0108 18:44:25.153316 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 24782707 bytes on disk I0108 18:44:25.153414 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4} I0108 18:44:25.265691 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 5.641s user 1.553s sys 0.181s Metrics: {"bytes_written":21130585,"cfile_cache_hit":53,"cfile_cache_hit_bytes":6265172,"cfile_cache_miss":2805,"cfile_cache_miss_bytes":78692552,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":618,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":243907,"drs_written":1,"lbm_read_time_us":587736,"lbm_reads_1-10_ms":37,"lbm_reads_10-100_ms":8,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":2795,"lbm_write_time_us":189611,"lbm_writes_1-10_ms":33,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":2109,"num_input_rowsets":2,"rows_written":1207636,"spinlock_wait_cycles":74377728,"thread_start_us":486,"threads_started":9} I0108 18:44:25.265877 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 24760960 bytes on disk I0108 18:44:25.266010 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 150000 Mean: 1897.73 Percentiles: 0% (min) = 804 25% = 1441 50% (med) = 1623 75% = 1871 95% = 2574 99% = 7376 99.9% = 44096 99.99% = 67456 100% (max) = 147792 I0108 18:44:28.539039 24709 linked_list-test-util.h:519] Written 7500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 160000 Mean: 1899.88 Percentiles: 0% (min) = 804 25% = 1445 50% (med) = 1626 75% = 1872 95% = 2564 99% = 7380 99.9% = 44192 99.99% = 68352 100% (max) = 147792 I0108 18:44:49.498971 24709 linked_list-test-util.h:519] Written 8000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 170000 Mean: 1901.42 Percentiles: 0% (min) = 804 25% = 1449 50% (med) = 1630 75% = 1876 95% = 2556 99% = 7372 99.9% = 43648 99.99% = 67456 100% (max) = 147792 I0108 18:45:10.502408 24709 linked_list-test-util.h:519] Written 8500000 rows in chain I0108 18:45:25.957643 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033380 I0108 18:45:27.358574 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 1.401s user 0.626s sys 0.032s Metrics: {"bytes_written":8029668,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1196,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":40672,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":39816,"lbm_writes_1-10_ms":5,"lbm_writes_lt_1ms":815,"reinsert_count":0,"rows_written":453362,"thread_start_us":32,"threads_started":1,"update_count":286} I0108 18:45:27.358671 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41977826 bytes of WAL I0108 18:45:27.358705 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:45:27.358711 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000010 (ops 111516-123869) I0108 18:45:27.358741 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000011 (ops 123870-136313) I0108 18:45:27.358901 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000012 (ops 136314-148661) I0108 18:45:27.359036 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000013 (ops 148662-161055) I0108 18:45:27.359438 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000014 (ops 161056-173454) I0108 18:45:27.377910 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.019s user 0.002s sys 0.010s Metrics: {} I0108 18:45:27.377997 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 9002010 bytes on disk I0108 18:45:27.378103 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:45:27.378150 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.001430 I0108 18:45:31.914672 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 4.536s user 1.872s sys 0.173s Metrics: {"bytes_written":26034515,"cfile_cache_hit":69,"cfile_cache_hit_bytes":8785919,"cfile_cache_miss":3421,"cfile_cache_miss_bytes":95801631,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":9237,"dirs.run_cpu_time_us":100,"dirs.run_wall_time_us":33728,"drs_written":1,"lbm_read_time_us":313605,"lbm_reads_1-10_ms":29,"lbm_reads_10-100_ms":9,"lbm_reads_lt_1ms":3415,"lbm_write_time_us":148007,"lbm_writes_1-10_ms":16,"lbm_writes_lt_1ms":2607,"num_input_rowsets":2,"rows_written":1478654,"spinlock_wait_cycles":15923712,"thread_start_us":511,"threads_started":9} I0108 18:45:31.914855 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.033745 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 180000 Mean: 1906.99 Percentiles: 0% (min) = 804 25% = 1454 50% (med) = 1637 75% = 1884 95% = 2558 99% = 7376 99.9% = 43648 99.99% = 67456 100% (max) = 147792 I0108 18:45:32.131222 24709 linked_list-test-util.h:519] Written 9000000 rows in chain I0108 18:45:33.652627 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 1.738s user 0.670s sys 0.053s Metrics: {"bytes_written":8148444,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":29805,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":105853,"lbm_writes_1-10_ms":11,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":815,"reinsert_count":0,"rows_written":457709,"thread_start_us":27,"threads_started":1,"update_count":3702} I0108 18:45:33.652735 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41999322 bytes of WAL I0108 18:45:33.652803 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:45:33.652818 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000010 (ops 111932-124342) I0108 18:45:33.652856 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000011 (ops 124343-136828) I0108 18:45:33.652889 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000012 (ops 136829-149309) I0108 18:45:33.653165 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000013 (ops 149310-161700) I0108 18:45:33.653328 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000014 (ops 161701-174132) I0108 18:45:33.675854 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.023s user 0.001s sys 0.011s Metrics: {} I0108 18:45:33.676028 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 30588830 bytes on disk I0108 18:45:33.693248 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.017s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16732,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":2} I0108 18:45:33.693439 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 9066472 bytes on disk I0108 18:45:33.693713 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 18:45:33.693821 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.001382 I0108 18:45:38.471684 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 4.778s user 1.872s sys 0.187s Metrics: {"bytes_written":26468101,"cfile_cache_hit":64,"cfile_cache_hit_bytes":8177456,"cfile_cache_miss":3472,"cfile_cache_miss_bytes":97962858,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":13231,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":34097,"drs_written":1,"lbm_read_time_us":412554,"lbm_reads_1-10_ms":38,"lbm_reads_10-100_ms":9,"lbm_reads_lt_1ms":3457,"lbm_write_time_us":123220,"lbm_writes_1-10_ms":8,"lbm_writes_lt_1ms":2650,"num_input_rowsets":2,"rows_written":1499169,"spinlock_wait_cycles":1974144,"thread_start_us":869,"threads_started":9} I0108 18:45:38.471783 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 31054142 bytes on disk I0108 18:45:38.471870 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 18:45:45.843097 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033341 I0108 18:45:46.974545 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033352 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 190000 Mean: 1916.37 Percentiles: 0% (min) = 804 25% = 1458 50% (med) = 1642 75% = 1892 95% = 2576 99% = 7444 99.9% = 44096 99.99% = 67456 100% (max) = 147792 I0108 18:45:54.425307 24709 linked_list-test-util.h:519] Written 9500000 rows in chain I0108 18:45:54.484282 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033391 I0108 18:45:55.294661 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 9.451s user 3.125s sys 0.226s Metrics: {"bytes_written":34803077,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":7388,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":52276,"drs_written":2,"lbm_read_time_us":19495,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":7,"lbm_write_time_us":250278,"lbm_writes_1-10_ms":39,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3457,"reinsert_count":0,"rows_written":1960313,"thread_start_us":31,"threads_started":1,"update_count":11897} I0108 18:45:55.294864 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 83902325 bytes of WAL I0108 18:45:55.294939 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:45:55.294986 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000022 (ops 123955-129775) I0108 18:45:55.295135 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000023 (ops 129776-135730) I0108 18:45:55.295212 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000024 (ops 135731-141636) I0108 18:45:55.295281 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000025 (ops 141637-147476) I0108 18:45:55.295373 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000026 (ops 147477-153427) I0108 18:45:55.295439 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000027 (ops 153428-159355) I0108 18:45:55.295533 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000028 (ops 159356-165167) I0108 18:45:55.295596 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000029 (ops 165168-171045) I0108 18:45:55.295655 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000030 (ops 171046-176992) I0108 18:45:55.295718 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000031 (ops 176993-182889) I0108 18:45:55.340162 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.045s user 0.002s sys 0.020s Metrics: {} I0108 18:45:55.340358 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.010503 I0108 18:45:57.276361 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 10.302s user 3.242s sys 0.274s Metrics: {"bytes_written":35029162,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":133,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":120401,"drs_written":2,"lbm_read_time_us":70662,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":7,"lbm_write_time_us":276492,"lbm_writes_1-10_ms":60,"lbm_writes_lt_1ms":3461,"reinsert_count":0,"rows_written":1972638,"spinlock_wait_cycles":1280,"thread_start_us":80,"threads_started":1,"update_count":12675} I0108 18:45:57.276569 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 84305229 bytes of WAL I0108 18:45:57.276643 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:45:57.276690 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000022 (ops 124278-130174) I0108 18:45:57.276780 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000023 (ops 130175-136100) I0108 18:45:57.276849 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000024 (ops 136101-142049) I0108 18:45:57.276912 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000025 (ops 142050-147982) I0108 18:45:57.276978 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000026 (ops 147983-153934) I0108 18:45:57.277040 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000027 (ops 153935-159838) I0108 18:45:57.277132 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000028 (ops 159839-165778) I0108 18:45:57.277206 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000029 (ops 165779-171702) I0108 18:45:57.277271 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000030 (ops 171703-177635) I0108 18:45:57.277354 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000031 (ops 177636-183556) I0108 18:45:57.317138 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.040s user 0.000s sys 0.023s Metrics: {} I0108 18:45:57.317328 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 39240152 bytes on disk I0108 18:45:57.317476 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":20,"lbm_reads_lt_1ms":8} I0108 18:45:57.317559 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.010331 I0108 18:46:04.908166 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 10.424s user 3.227s sys 0.206s Metrics: {"bytes_written":35137249,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":11824,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":66413,"drs_written":2,"lbm_read_time_us":10934,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":7,"lbm_write_time_us":362432,"lbm_writes_1-10_ms":74,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":3463,"reinsert_count":0,"rows_written":1984612,"spinlock_wait_cycles":62848,"thread_start_us":48,"threads_started":1,"update_count":3246} I0108 18:46:04.908393 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.009237 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 200000 Mean: 1936.82 Percentiles: 0% (min) = 804 25% = 1463 50% (med) = 1651 75% = 1909 95% = 2646 99% = 7564 99.9% = 44992 99.99% = 69184 100% (max) = 147792 I0108 18:46:18.860798 24709 linked_list-test-util.h:519] Written 10000000 rows in chain I0108 18:46:30.516481 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 35.173s user 8.419s sys 0.889s Metrics: {"bytes_written":110087197,"cfile_cache_hit":327,"cfile_cache_hit_bytes":45570758,"cfile_cache_miss":14321,"cfile_cache_miss_bytes":394562458,"cfile_init":23,"delta_iterators_relevant":14,"dirs.queue_time_us":12412,"dirs.run_cpu_time_us":492,"dirs.run_wall_time_us":149139,"drs_written":4,"lbm_read_time_us":3869677,"lbm_reads_1-10_ms":257,"lbm_reads_10-100_ms":59,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":14091,"lbm_write_time_us":1208005,"lbm_writes_1-10_ms":254,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":10748,"num_input_rowsets":5,"rows_written":6209563,"spinlock_wait_cycles":409532032,"thread_start_us":173,"threads_started":4} I0108 18:46:30.516666 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 129089453 bytes on disk I0108 18:46:30.516870 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":65,"lbm_reads_lt_1ms":16} I0108 18:46:30.516963 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.038602 I0108 18:46:31.118423 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 33.801s user 8.368s sys 0.846s Metrics: {"bytes_written":110447880,"cfile_cache_hit":326,"cfile_cache_hit_bytes":45474855,"cfile_cache_miss":14364,"cfile_cache_miss_bytes":395838359,"cfile_init":21,"delta_iterators_relevant":14,"dirs.queue_time_us":18278,"dirs.run_cpu_time_us":441,"dirs.run_wall_time_us":222080,"drs_written":4,"lbm_read_time_us":3467670,"lbm_reads_1-10_ms":237,"lbm_reads_10-100_ms":52,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":14153,"lbm_write_time_us":1089720,"lbm_writes_1-10_ms":228,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":10808,"num_input_rowsets":5,"rows_written":6226465,"spinlock_wait_cycles":90712832,"thread_start_us":631,"threads_started":12} I0108 18:46:31.118651 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.038635 I0108 18:46:33.256635 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.740s user 0.805s sys 0.057s Metrics: {"bytes_written":9308622,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":6602,"dirs.run_cpu_time_us":52,"dirs.run_wall_time_us":52,"drs_written":1,"lbm_read_time_us":21,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756504,"lbm_writes_1-10_ms":30,"lbm_writes_10-100_ms":18,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":899,"reinsert_count":0,"rows_written":524978,"thread_start_us":31,"threads_started":1,"update_count":2397} I0108 18:46:33.258627 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41944298 bytes of WAL I0108 18:46:33.258713 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:46:33.258769 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000012 (ops 136528-148857) I0108 18:46:33.258857 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000013 (ops 148858-161269) I0108 18:46:33.258940 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000014 (ops 161270-173669) I0108 18:46:33.259018 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000015 (ops 173670-186064) I0108 18:46:33.259105 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000016 (ops 186065-198461) I0108 18:46:33.272785 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.014s user 0.000s sys 0.013s Metrics: {} I0108 18:46:33.272965 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.000697 I0108 18:46:34.453197 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 3.334s user 0.798s sys 0.059s Metrics: {"bytes_written":9270745,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":2703,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":53329,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899626,"lbm_writes_1-10_ms":42,"lbm_writes_10-100_ms":22,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":882,"reinsert_count":0,"rows_written":524559,"thread_start_us":34,"threads_started":1,"update_count":60} I0108 18:46:34.453404 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 42054283 bytes of WAL I0108 18:46:34.453486 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:46:34.453534 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000012 (ops 136829-149309) I0108 18:46:34.453603 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000013 (ops 149310-161677) I0108 18:46:34.453660 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000014 (ops 161678-174104) I0108 18:46:34.453716 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000015 (ops 174105-186695) I0108 18:46:34.453773 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000016 (ops 186696-199092) I0108 18:46:34.469118 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.016s user 0.001s sys 0.012s Metrics: {} I0108 18:46:34.469337 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.000704 I0108 18:46:37.774065 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 32.866s user 8.521s sys 0.852s Metrics: {"bytes_written":112726568,"cfile_cache_hit":328,"cfile_cache_hit_bytes":45818944,"cfile_cache_miss":14614,"cfile_cache_miss_bytes":403227406,"cfile_init":23,"delta_iterators_relevant":14,"dirs.queue_time_us":17820,"dirs.run_cpu_time_us":422,"dirs.run_wall_time_us":155638,"drs_written":4,"lbm_read_time_us":3638023,"lbm_reads_1-10_ms":239,"lbm_reads_10-100_ms":58,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":14401,"lbm_write_time_us":1006468,"lbm_writes_1-10_ms":215,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":11011,"num_input_rowsets":5,"rows_written":6333845,"spinlock_wait_cycles":159971968,"thread_start_us":853,"threads_started":12} I0108 18:46:37.774271 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 84041644 bytes of WAL I0108 18:46:37.774598 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:46:37.774662 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000023 (ops 130555-136449) I0108 18:46:37.774745 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000024 (ops 136450-142318) I0108 18:46:37.774817 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000025 (ops 142319-148284) I0108 18:46:37.774926 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000026 (ops 148285-154114) I0108 18:46:37.774996 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000027 (ops 154115-160066) I0108 18:46:37.775063 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000028 (ops 160067-166023) I0108 18:46:37.775132 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000029 (ops 166024-171920) I0108 18:46:37.775199 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000030 (ops 171921-177873) I0108 18:46:37.775315 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000031 (ops 177874-183791) I0108 18:46:37.775383 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000032 (ops 183792-189682) I0108 18:46:37.829638 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.055s user 0.002s sys 0.023s Metrics: {} I0108 18:46:37.829908 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 131825461 bytes on disk I0108 18:46:38.084312 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.254s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":254068,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":6,"lbm_reads_lt_1ms":8} I0108 18:46:41.640307 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 8.367s user 2.252s sys 0.221s Metrics: {"bytes_written":30427080,"cfile_cache_hit":72,"cfile_cache_hit_bytes":9411761,"cfile_cache_miss":3962,"cfile_cache_miss_bytes":111755627,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":15208,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":37244,"drs_written":1,"lbm_read_time_us":987432,"lbm_reads_1-10_ms":67,"lbm_reads_10-100_ms":21,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3909,"lbm_write_time_us":270436,"lbm_writes_1-10_ms":51,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2974,"mutex_wait_us":116,"num_input_rowsets":2,"rows_written":1708024,"spinlock_wait_cycles":9216,"thread_start_us":830,"threads_started":9} I0108 18:46:41.640477 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 35656221 bytes on disk I0108 18:46:41.640610 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0108 18:46:41.640712 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.039481 I0108 18:46:42.533869 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 8.064s user 2.194s sys 0.255s Metrics: {"bytes_written":30405973,"cfile_cache_hit":77,"cfile_cache_hit_bytes":10226360,"cfile_cache_miss":3955,"cfile_cache_miss_bytes":110843880,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":9516,"dirs.run_cpu_time_us":96,"dirs.run_wall_time_us":37112,"drs_written":1,"lbm_read_time_us":1359955,"lbm_reads_1-10_ms":71,"lbm_reads_10-100_ms":20,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":3898,"lbm_write_time_us":257574,"lbm_writes_1-10_ms":42,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2981,"num_input_rowsets":2,"rows_written":1706705,"spinlock_wait_cycles":11624576,"thread_start_us":1057,"threads_started":9} I0108 18:46:42.534031 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.039703 I0108 18:46:43.869033 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.228s user 0.776s sys 0.049s Metrics: {"bytes_written":9447810,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":80,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":39876,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":174082,"lbm_writes_1-10_ms":13,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":945,"reinsert_count":0,"rows_written":533152,"thread_start_us":38,"threads_started":1,"update_count":2487} I0108 18:46:43.869205 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41972419 bytes of WAL I0108 18:46:43.869345 24932 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:46:43.869390 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000012 (ops 136363-148802) I0108 18:46:43.869454 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000013 (ops 148803-161165) I0108 18:46:43.869556 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000014 (ops 161166-173569) I0108 18:46:43.869657 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000015 (ops 173570-185973) I0108 18:46:43.869715 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000016 (ops 185974-198384) I0108 18:46:43.944525 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.075s user 0.002s sys 0.012s Metrics: {} I0108 18:46:43.944692 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.000610 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 210000 Mean: 1959.73 Percentiles: 0% (min) = 804 25% = 1468 50% (med) = 1658 75% = 1921 95% = 2686 99% = 7800 99.9% = 46528 99.99% = 75520 100% (max) = 191923 I0108 18:46:44.331797 24709 linked_list-test-util.h:519] Written 10500000 rows in chain I0108 18:46:44.369494 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 1.835s user 0.712s sys 0.048s Metrics: {"bytes_written":9495202,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":48727,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":114425,"lbm_writes_1-10_ms":12,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":952,"reinsert_count":0,"rows_written":536364,"thread_start_us":32,"threads_started":1,"update_count":1621} I0108 18:46:44.369922 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41949567 bytes of WAL I0108 18:46:44.370090 24807 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:46:44.370146 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000012 (ops 136546-148913) I0108 18:46:44.370295 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000013 (ops 148914-161330) I0108 18:46:44.370445 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000014 (ops 161331-173738) I0108 18:46:44.370512 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000015 (ops 173739-186140) I0108 18:46:44.370581 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000016 (ops 186141-198482) I0108 18:46:44.416573 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.046s user 0.001s sys 0.014s Metrics: {} I0108 18:46:44.417304 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 129460732 bytes on disk I0108 18:46:44.490360 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.073s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":72628,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":8} I0108 18:46:44.490500 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.000576 I0108 18:46:50.195937 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 6.251s user 2.291s sys 0.242s Metrics: {"bytes_written":30995139,"cfile_cache_hit":74,"cfile_cache_hit_bytes":9664884,"cfile_cache_miss":4032,"cfile_cache_miss_bytes":113912176,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":57,"dirs.run_wall_time_us":57,"drs_written":1,"lbm_read_time_us":328474,"lbm_reads_1-10_ms":29,"lbm_reads_10-100_ms":8,"lbm_reads_lt_1ms":4031,"lbm_write_time_us":209117,"lbm_writes_1-10_ms":23,"lbm_writes_lt_1ms":3062,"num_input_rowsets":2,"rows_written":1740788,"spinlock_wait_cycles":48021376,"thread_start_us":673,"threads_started":9} I0108 18:46:50.196111 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 36402561 bytes on disk I0108 18:46:50.196233 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:46:50.733031 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 6.242s user 2.217s sys 0.226s Metrics: {"bytes_written":31068981,"cfile_cache_hit":82,"cfile_cache_hit_bytes":10930319,"cfile_cache_miss":4032,"cfile_cache_miss_bytes":112959367,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":3076,"dirs.run_cpu_time_us":58,"dirs.run_wall_time_us":58,"drs_written":1,"lbm_read_time_us":840018,"lbm_reads_1-10_ms":29,"lbm_reads_10-100_ms":16,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":4021,"lbm_write_time_us":224427,"lbm_writes_1-10_ms":27,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3063,"num_input_rowsets":2,"rows_written":1745306,"spinlock_wait_cycles":820480,"thread_start_us":2260,"threads_started":9} I0108 18:46:50.733206 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 36494538 bytes on disk I0108 18:46:50.733397 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 18:46:50.733630 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 35631280 bytes on disk I0108 18:46:50.804163 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.070s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70243,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":1} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 220000 Mean: 1964.64 Percentiles: 0% (min) = 804 25% = 1471 50% (med) = 1661 75% = 1925 95% = 2690 99% = 7816 99.9% = 46784 99.99% = 76736 100% (max) = 228066 I0108 18:47:06.370440 24709 linked_list-test-util.h:519] Written 11000000 rows in chain I0108 18:47:27.380733 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033339 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 230000 Mean: 1964.36 Percentiles: 0% (min) = 804 25% = 1473 50% (med) = 1663 75% = 1925 95% = 2678 99% = 7820 99.9% = 46720 99.99% = 75520 100% (max) = 228066 I0108 18:47:27.588438 24709 linked_list-test-util.h:519] Written 11500000 rows in chain I0108 18:47:28.662453 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 1.282s user 0.618s sys 0.038s Metrics: {"bytes_written":7686424,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":16416,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":31932,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":35128,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":791,"reinsert_count":0,"rows_written":438653,"thread_start_us":30,"threads_started":1,"update_count":2048} I0108 18:47:28.662633 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 33572605 bytes of WAL I0108 18:47:28.662712 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 4 log segments from log reader I0108 18:47:28.662757 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000015 (ops 173455-185849) I0108 18:47:28.662829 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000016 (ops 185850-198243) I0108 18:47:28.662896 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000017 (ops 198244-210617) I0108 18:47:28.662963 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000018 (ops 210618-223004) I0108 18:47:28.683598 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.021s user 0.000s sys 0.012s Metrics: {} I0108 18:47:28.683749 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.001601 I0108 18:47:34.849454 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 6.166s user 2.368s sys 0.245s Metrics: {"bytes_written":34083231,"cfile_cache_hit":87,"cfile_cache_hit_bytes":11327526,"cfile_cache_miss":4441,"cfile_cache_miss_bytes":125449392,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":468,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":102778,"drs_written":2,"lbm_read_time_us":456135,"lbm_reads_1-10_ms":31,"lbm_reads_10-100_ms":14,"lbm_reads_lt_1ms":4436,"lbm_write_time_us":297362,"lbm_writes_1-10_ms":22,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3393,"num_input_rowsets":2,"rows_written":1917307,"spinlock_wait_cycles":1379840,"thread_start_us":709,"threads_started":9} I0108 18:47:34.849619 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 40271080 bytes on disk I0108 18:47:34.849746 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8} I0108 18:47:34.849833 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.033666 I0108 18:47:36.480548 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 1.631s user 0.642s sys 0.051s Metrics: {"bytes_written":7788715,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":5524,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":34851,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":142294,"lbm_writes_1-10_ms":16,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":784,"reinsert_count":0,"rows_written":443235,"thread_start_us":31,"threads_started":1,"update_count":3267} I0108 18:47:36.480718 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33652698 bytes of WAL I0108 18:47:36.480813 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:47:36.480865 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000015 (ops 174133-186697) I0108 18:47:36.480962 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000016 (ops 186698-199068) I0108 18:47:36.481055 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000017 (ops 199069-211514) I0108 18:47:36.481122 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000018 (ops 211515-223909) I0108 18:47:36.499538 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.019s user 0.001s sys 0.010s Metrics: {} I0108 18:47:36.499688 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 8768572 bytes on disk I0108 18:47:36.499816 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 18:47:36.499912 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.001558 I0108 18:47:42.810473 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 6.310s user 2.438s sys 0.248s Metrics: {"bytes_written":34523619,"cfile_cache_hit":86,"cfile_cache_hit_bytes":11137196,"cfile_cache_miss":4496,"cfile_cache_miss_bytes":127649976,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":1056,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":51901,"drs_written":2,"lbm_read_time_us":472038,"lbm_reads_1-10_ms":53,"lbm_reads_10-100_ms":8,"lbm_reads_lt_1ms":4471,"lbm_write_time_us":245119,"lbm_writes_1-10_ms":33,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3427,"num_input_rowsets":2,"rows_written":1942404,"spinlock_wait_cycles":351104,"thread_start_us":495,"threads_started":9} I0108 18:47:42.810662 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 40815050 bytes on disk I0108 18:47:42.953271 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.142s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":142395,"lbm_reads_1-10_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 240000 Mean: 1969.81 Percentiles: 0% (min) = 804 25% = 1476 50% (med) = 1666 75% = 1930 95% = 2684 99% = 7844 99.9% = 46816 99.99% = 75520 100% (max) = 228066 I0108 18:47:50.073424 24709 linked_list-test-util.h:519] Written 12000000 rows in chain I0108 18:47:55.509546 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033393 I0108 18:47:57.367354 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033359 I0108 18:48:03.134961 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 7.625s user 2.897s sys 0.199s Metrics: {"bytes_written":33007958,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":119,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":41076,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":191937,"lbm_writes_1-10_ms":24,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3300,"reinsert_count":0,"rows_written":1875777,"thread_start_us":83,"threads_started":1,"update_count":3756} I0108 18:48:03.135202 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 83912414 bytes of WAL I0108 18:48:03.135283 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:48:03.135350 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000032 (ops 182890-188711) I0108 18:48:03.135489 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000033 (ops 188712-194584) I0108 18:48:03.135560 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000034 (ops 194585-200492) I0108 18:48:03.135625 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000035 (ops 200493-206387) I0108 18:48:03.135687 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000036 (ops 206388-212337) I0108 18:48:03.135846 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000037 (ops 212338-218182) I0108 18:48:03.136023 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000038 (ops 218183-224110) I0108 18:48:03.136330 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000039 (ops 224111-230037) I0108 18:48:03.136400 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000040 (ops 230038-235891) I0108 18:48:03.136466 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000041 (ops 235892-241867) I0108 18:48:03.186164 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.051s user 0.000s sys 0.028s Metrics: {} I0108 18:48:03.186434 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 37364306 bytes on disk I0108 18:48:03.186563 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0108 18:48:03.186688 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.000754 I0108 18:48:04.977576 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033353 I0108 18:48:06.191067 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 8.824s user 2.908s sys 0.174s Metrics: {"bytes_written":33139236,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":302,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":28611,"drs_written":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4,"lbm_write_time_us":347658,"lbm_writes_1-10_ms":40,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3300,"reinsert_count":0,"rows_written":1884852,"spinlock_wait_cycles":23296,"thread_start_us":29,"threads_started":1,"update_count":1262} I0108 18:48:06.191282 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 37566892 bytes on disk I0108 18:48:06.191413 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0108 18:48:06.191519 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=1.000578 I0108 18:48:13.423748 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 8.446s user 2.875s sys 0.163s Metrics: {"bytes_written":33196811,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15514,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":61803,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":231392,"lbm_writes_1-10_ms":48,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3289,"reinsert_count":0,"rows_written":1882981,"thread_start_us":105,"threads_started":1,"update_count":9158} I0108 18:48:13.424554 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 75800734 bytes of WAL I0108 18:48:13.424639 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 9 log segments from log reader I0108 18:48:13.424690 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000033 (ops 189683-195539) I0108 18:48:13.424768 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000034 (ops 195540-201575) I0108 18:48:13.424968 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000035 (ops 201576-207419) I0108 18:48:13.425043 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000036 (ops 207420-213276) I0108 18:48:13.425115 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000037 (ops 213277-219203) I0108 18:48:13.425319 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000038 (ops 219204-225156) I0108 18:48:13.425411 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000039 (ops 225157-231097) I0108 18:48:13.425477 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000040 (ops 231098-237054) I0108 18:48:13.425541 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000041 (ops 237055-242990) I0108 18:48:13.469210 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.045s user 0.000s sys 0.024s Metrics: {} I0108 18:48:13.469379 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 37445911 bytes on disk I0108 18:48:13.470448 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 18:48:13.470549 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.999454 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 250000 Mean: 1980.34 Percentiles: 0% (min) = 804 25% = 1479 50% (med) = 1671 75% = 1939 95% = 2712 99% = 7924 99.9% = 47040 99.99% = 77248 100% (max) = 275150 I0108 18:48:13.888988 24709 linked_list-test-util.h:519] Written 12500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 260000 Mean: 1996.55 Percentiles: 0% (min) = 804 25% = 1484 50% (med) = 1679 75% = 1956 95% = 2762 99% = 7996 99.9% = 47200 99.99% = 78848 100% (max) = 275150 I0108 18:48:39.140316 24709 linked_list-test-util.h:519] Written 13000000 rows in chain I0108 18:48:43.853410 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 40.667s user 10.882s sys 1.074s Metrics: {"bytes_written":143612573,"cfile_cache_hit":364,"cfile_cache_hit_bytes":50684057,"cfile_cache_miss":18692,"cfile_cache_miss_bytes":529427917,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":27486,"dirs.run_cpu_time_us":541,"dirs.run_wall_time_us":421799,"drs_written":5,"lbm_read_time_us":5118657,"lbm_reads_1-10_ms":318,"lbm_reads_10-100_ms":74,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":18374,"lbm_write_time_us":1244142,"lbm_writes_1-10_ms":220,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":14104,"num_input_rowsets":5,"rows_written":8085340,"spinlock_wait_cycles":296912128,"thread_start_us":3619,"threads_started":13} I0108 18:48:43.853966 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.036277 I0108 18:48:45.084416 28042 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.66:58364 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=776354, attempt_no=0}) took 1073ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":6659,"cfile_cache_hit":848,"cfile_cache_hit_bytes":3524336,"cfile_cache_miss":3533,"cfile_cache_miss_bytes":14795663,"lbm_read_time_us":922366,"lbm_reads_1-10_ms":33,"lbm_reads_10-100_ms":31,"lbm_reads_lt_1ms":3469,"num_ops":11858,"prepare.queue_time_us":11,"prepare.run_cpu_time_us":12492,"prepare.run_wall_time_us":30414,"raft.queue_time_us":90,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"replication_time_us":1682,"spinlock_wait_cycles":3328}]]} I0108 18:48:45.531427 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 39.340s user 10.571s sys 1.012s Metrics: {"bytes_written":144069713,"cfile_cache_hit":377,"cfile_cache_hit_bytes":52308025,"cfile_cache_miss":18741,"cfile_cache_miss_bytes":529803655,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":7977,"dirs.run_cpu_time_us":609,"dirs.run_wall_time_us":194386,"drs_written":5,"lbm_read_time_us":5302195,"lbm_reads_1-10_ms":286,"lbm_reads_10-100_ms":71,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":18458,"lbm_write_time_us":1263725,"lbm_writes_1-10_ms":237,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":14133,"num_input_rowsets":5,"rows_written":8111317,"spinlock_wait_cycles":121716224,"thread_start_us":845,"threads_started":13} I0108 18:48:45.531637 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 84171932 bytes of WAL I0108 18:48:45.531723 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 10 log segments from log reader I0108 18:48:45.531735 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000032 (ops 183557-189436) I0108 18:48:45.531772 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000033 (ops 189437-195405) I0108 18:48:45.531801 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000034 (ops 195406-201508) I0108 18:48:45.531826 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000035 (ops 201509-207425) I0108 18:48:45.531847 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000036 (ops 207426-213269) I0108 18:48:45.531868 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000037 (ops 213270-219175) I0108 18:48:45.531890 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000038 (ops 219176-225116) I0108 18:48:45.531913 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000039 (ops 225117-231055) I0108 18:48:45.531956 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000040 (ops 231056-236884) I0108 18:48:45.531977 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000041 (ops 236885-242844) I0108 18:48:45.552439 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.021s user 0.000s sys 0.021s Metrics: {} I0108 18:48:45.552564 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 171222975 bytes on disk I0108 18:48:46.052731 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.500s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":499853,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":4,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":10} I0108 18:48:46.053002 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.036555 I0108 18:48:46.414417 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.560s user 0.698s sys 0.040s Metrics: {"bytes_written":8398088,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13617,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":28087,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1046799,"lbm_writes_1-10_ms":18,"lbm_writes_10-100_ms":30,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":810,"reinsert_count":0,"rows_written":475975,"thread_start_us":69,"threads_started":1,"update_count":2801} I0108 18:48:46.414587 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 33559145 bytes of WAL I0108 18:48:46.414660 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 4 log segments from log reader I0108 18:48:46.414758 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000017 (ops 198462-210831) I0108 18:48:46.415874 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000018 (ops 210832-223219) I0108 18:48:46.415948 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000019 (ops 223220-235615) I0108 18:48:46.416144 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000020 (ops 235616-248007) I0108 18:48:46.434193 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.020s user 0.003s sys 0.008s Metrics: {} I0108 18:48:46.434370 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.001213 I0108 18:48:48.764430 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.711s user 0.712s sys 0.062s Metrics: {"bytes_written":8421841,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":161,"dirs.run_cpu_time_us":88,"dirs.run_wall_time_us":31076,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":635639,"lbm_writes_1-10_ms":36,"lbm_writes_10-100_ms":22,"lbm_writes_lt_1ms":805,"reinsert_count":0,"rows_written":478155,"thread_start_us":32,"threads_started":1,"update_count":1553} I0108 18:48:48.764600 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33573636 bytes of WAL I0108 18:48:48.764679 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:48:48.764731 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000017 (ops 199093-211554) I0108 18:48:48.764801 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000018 (ops 211555-223909) I0108 18:48:48.764894 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000019 (ops 223910-236372) I0108 18:48:48.765035 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000020 (ops 236373-248751) I0108 18:48:48.814287 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.050s user 0.000s sys 0.013s Metrics: {} I0108 18:48:48.815440 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 9491560 bytes on disk I0108 18:48:48.815574 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":18,"lbm_reads_lt_1ms":4} I0108 18:48:48.815682 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.001195 I0108 18:48:55.334699 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 41.864s user 10.953s sys 1.025s Metrics: {"bytes_written":145915496,"cfile_cache_hit":369,"cfile_cache_hit_bytes":51629030,"cfile_cache_miss":18997,"cfile_cache_miss_bytes":539131710,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":12668,"dirs.run_cpu_time_us":672,"dirs.run_wall_time_us":519936,"drs_written":5,"lbm_read_time_us":5970778,"lbm_reads_1-10_ms":287,"lbm_reads_10-100_ms":104,"lbm_reads_gt_100_ms":12,"lbm_reads_lt_1ms":18678,"lbm_write_time_us":1465660,"lbm_writes_1-10_ms":236,"lbm_writes_10-100_ms":14,"lbm_writes_lt_1ms":14314,"num_input_rowsets":5,"rows_written":8216826,"spinlock_wait_cycles":260304384,"thread_start_us":536,"threads_started":13} I0108 18:48:55.334987 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 173559944 bytes on disk I0108 18:48:55.335150 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":29,"lbm_reads_lt_1ms":20} I0108 18:48:56.030304 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 9.596s user 2.769s sys 0.247s Metrics: {"bytes_written":38756672,"cfile_cache_hit":103,"cfile_cache_hit_bytes":13749639,"cfile_cache_miss":5051,"cfile_cache_miss_bytes":143565905,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":458,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":115500,"drs_written":2,"lbm_read_time_us":1067265,"lbm_reads_1-10_ms":68,"lbm_reads_10-100_ms":24,"lbm_reads_lt_1ms":4999,"lbm_write_time_us":377728,"lbm_writes_1-10_ms":69,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3809,"num_input_rowsets":2,"rows_written":2183999,"spinlock_wait_cycles":22036352,"thread_start_us":488,"threads_started":9} I0108 18:48:56.035423 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 170657080 bytes on disk I0108 18:48:56.790489 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.755s user 0.000s sys 0.001s Metrics: {"cfile_init":5,"lbm_read_time_us":754681,"lbm_reads_10-100_ms":6,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":10} I0108 18:48:56.790611 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 46205384 bytes on disk I0108 18:48:56.877665 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.087s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":86867,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":4} I0108 18:48:56.877797 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.036947 W0108 18:48:57.265548 28160 lock_manager.cc:368] Waited 1 seconds to obtain row lock on key \xc7\xf8\x1f\x1e\x20\xbb\x00\x05 cur holder: 0x5d67180 I0108 18:48:57.813882 28140 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.67:45792 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=788079, attempt_no=0}) took 2453ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":2452725,"cfile_cache_hit":8,"cfile_cache_hit_bytes":44340,"num_ops":7,"prepare.queue_time_us":9,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":58,"raft.run_cpu_time_us":43,"raft.run_wall_time_us":44,"replication_time_us":614,"spinlock_wait_cycles":11008,"thread_start_us":32,"threads_started":1}]]} I0108 18:48:57.824700 28127 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.67:45792 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=788083, attempt_no=0}) took 1562ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1560527,"cfile_cache_hit":89,"cfile_cache_hit_bytes":372959,"num_ops":98,"prepare.queue_time_us":72,"prepare.run_cpu_time_us":270,"prepare.run_wall_time_us":450,"raft.queue_time_us":32,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":116,"replication_time_us":741,"spinlock_wait_cycles":130176,"thread_start_us":33,"threads_started":1}]]} I0108 18:48:57.982199 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 9.166s user 2.574s sys 0.254s Metrics: {"bytes_written":38775349,"cfile_cache_hit":95,"cfile_cache_hit_bytes":12501370,"cfile_cache_miss":5059,"cfile_cache_miss_bytes":144857540,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":638,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":75879,"drs_written":2,"lbm_read_time_us":1961519,"lbm_reads_1-10_ms":55,"lbm_reads_10-100_ms":58,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":4979,"lbm_write_time_us":329865,"lbm_writes_1-10_ms":65,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3818,"num_input_rowsets":2,"rows_written":2184860,"spinlock_wait_cycles":5494400,"thread_start_us":720,"threads_started":9} I0108 18:48:57.982409 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 46209192 bytes on disk I0108 18:48:58.059984 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.077s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":77409,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":6} I0108 18:48:58.060145 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.037136 I0108 18:48:59.512208 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.634s user 0.646s sys 0.046s Metrics: {"bytes_written":8351748,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":99,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":38552,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1228904,"lbm_writes_1-10_ms":27,"lbm_writes_10-100_ms":39,"lbm_writes_lt_1ms":793,"reinsert_count":0,"rows_written":475226,"spinlock_wait_cycles":5120,"thread_start_us":62,"threads_started":1,"update_count":9} I0108 18:48:59.512341 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41961238 bytes of WAL I0108 18:48:59.512432 24932 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:48:59.512485 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000017 (ops 198385-210762) I0108 18:48:59.512562 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000018 (ops 210763-223154) I0108 18:48:59.512622 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000019 (ops 223155-235523) I0108 18:48:59.512722 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000020 (ops 235524-247903) I0108 18:48:59.512802 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000021 (ops 247904-260321) I0108 18:48:59.538730 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.026s user 0.000s sys 0.013s Metrics: {} I0108 18:48:59.538877 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 9444510 bytes on disk I0108 18:48:59.539007 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4} I0108 18:48:59.539115 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.001226 I0108 18:48:59.714229 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 1.654s user 0.616s sys 0.043s Metrics: {"bytes_written":8303269,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":16596,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":11874,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":280383,"lbm_writes_1-10_ms":16,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":834,"reinsert_count":0,"rows_written":472141,"thread_start_us":31,"threads_started":1,"update_count":154} I0108 18:48:59.714351 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 42115862 bytes of WAL I0108 18:48:59.714428 24807 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:48:59.714443 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000017 (ops 198483-210915) I0108 18:48:59.714478 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000018 (ops 210916-223319) I0108 18:48:59.714511 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000019 (ops 223320-235726) I0108 18:48:59.714531 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000020 (ops 235727-248337) I0108 18:48:59.714637 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000021 (ops 248338-260725) I0108 18:48:59.734761 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.020s user 0.003s sys 0.010s Metrics: {} I0108 18:48:59.734915 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.001253 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 270000 Mean: 2024.71 Percentiles: 0% (min) = 804 25% = 1487 50% (med) = 1685 75% = 1967 95% = 2792 99% = 8164 99.9% = 48352 99.99% = 86784 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:49:08.013906 24709 linked_list-test-util.h:519] Written 13500000 rows in chain I0108 18:49:09.136937 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 9.598s user 2.909s sys 0.331s Metrics: {"bytes_written":39320709,"cfile_cache_hit":101,"cfile_cache_hit_bytes":13378186,"cfile_cache_miss":5129,"cfile_cache_miss_bytes":146318976,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":2381,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":105321,"drs_written":2,"lbm_read_time_us":1285019,"lbm_reads_1-10_ms":66,"lbm_reads_10-100_ms":28,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":5070,"lbm_write_time_us":347749,"lbm_writes_1-10_ms":30,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3908,"num_input_rowsets":2,"rows_written":2216014,"spinlock_wait_cycles":37447680,"thread_start_us":785,"threads_started":10} I0108 18:49:09.137125 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 46914787 bytes on disk I0108 18:49:09.141398 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 18:49:09.243624 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 9.509s user 2.942s sys 0.297s Metrics: {"bytes_written":39345433,"cfile_cache_hit":107,"cfile_cache_hit_bytes":14114371,"cfile_cache_miss":5125,"cfile_cache_miss_bytes":145696393,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":6863,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":145778,"drs_written":2,"lbm_read_time_us":1634689,"lbm_reads_1-10_ms":61,"lbm_reads_10-100_ms":36,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":5066,"lbm_write_time_us":351825,"lbm_writes_1-10_ms":42,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":3897,"num_input_rowsets":2,"rows_written":2217447,"spinlock_wait_cycles":2989952,"thread_start_us":907,"threads_started":10} I0108 18:49:09.243757 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 46967499 bytes on disk I0108 18:49:09.286598 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.043s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":42658,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":5} I0108 18:49:21.021355 24709 linked_list-test-util.h:528] Sample at HT timestamp: 6465692307533090816 Inserted count: 13799750 I0108 18:49:28.861536 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033389 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 280000 Mean: 2026.29 Percentiles: 0% (min) = 804 25% = 1488 50% (med) = 1686 75% = 1967 95% = 2792 99% = 8216 99.9% = 48448 99.99% = 85248 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:49:30.173285 24709 linked_list-test-util.h:519] Written 14000000 rows in chain I0108 18:49:30.802603 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 1.941s user 0.602s sys 0.049s Metrics: {"bytes_written":7277357,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":3817,"dirs.run_cpu_time_us":91,"dirs.run_wall_time_us":26196,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":61702,"lbm_writes_1-10_ms":13,"lbm_writes_lt_1ms":734,"reinsert_count":0,"rows_written":412092,"thread_start_us":33,"threads_started":1,"update_count":2374} I0108 18:49:30.802785 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 33561139 bytes of WAL I0108 18:49:30.802855 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 4 log segments from log reader I0108 18:49:30.802903 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000019 (ops 223005-235402) I0108 18:49:30.803198 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000020 (ops 235403-247772) I0108 18:49:30.803478 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000021 (ops 247773-260185) I0108 18:49:30.803712 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000022 (ops 260186-272595) I0108 18:49:30.822402 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.020s user 0.001s sys 0.010s Metrics: {} I0108 18:49:30.822551 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 8168216 bytes on disk I0108 18:49:30.822671 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0108 18:49:30.822755 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.010681 I0108 18:49:42.173393 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 11.351s user 3.169s sys 0.325s Metrics: {"bytes_written":41303239,"cfile_cache_hit":98,"cfile_cache_hit_bytes":12258264,"cfile_cache_miss":5410,"cfile_cache_miss_bytes":156105652,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":1309,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":135407,"drs_written":2,"lbm_read_time_us":746050,"lbm_reads_1-10_ms":101,"lbm_reads_10-100_ms":20,"lbm_reads_lt_1ms":5337,"lbm_write_time_us":452708,"lbm_writes_1-10_ms":91,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":4048,"num_input_rowsets":3,"rows_written":2329399,"spinlock_wait_cycles":25710336,"thread_start_us":519,"threads_started":10} I0108 18:49:42.173581 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.034915 I0108 18:49:44.349056 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.175s user 0.627s sys 0.059s Metrics: {"bytes_written":7619424,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":80855,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":336379,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":11,"lbm_writes_lt_1ms":755,"reinsert_count":0,"rows_written":434297,"thread_start_us":33,"threads_started":1,"update_count":2759} I0108 18:49:44.349484 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33579773 bytes of WAL I0108 18:49:44.349560 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:49:44.349612 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000019 (ops 223910-236374) I0108 18:49:44.349958 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000020 (ops 236375-248751) I0108 18:49:44.350028 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000021 (ops 248752-261190) I0108 18:49:44.350989 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000022 (ops 261191-273579) I0108 18:49:44.372787 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.023s user 0.000s sys 0.012s Metrics: {} I0108 18:49:44.372946 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.010203 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 290000 Mean: 2032.24 Percentiles: 0% (min) = 804 25% = 1490 50% (med) = 1688 75% = 1972 95% = 2800 99% = 8272 99.9% = 48672 99.99% = 86784 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:49:53.513453 24709 linked_list-test-util.h:519] Written 14500000 rows in chain I0108 18:49:54.023803 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 9.650s user 3.041s sys 0.337s Metrics: {"bytes_written":42128276,"cfile_cache_hit":115,"cfile_cache_hit_bytes":14526584,"cfile_cache_miss":5505,"cfile_cache_miss_bytes":157198978,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":11088,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":73233,"drs_written":2,"lbm_read_time_us":931066,"lbm_reads_1-10_ms":72,"lbm_reads_10-100_ms":23,"lbm_reads_lt_1ms":5462,"lbm_write_time_us":345126,"lbm_writes_1-10_ms":59,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":4162,"num_input_rowsets":3,"rows_written":2376701,"spinlock_wait_cycles":30767232,"thread_start_us":274,"threads_started":10} I0108 18:49:54.023972 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 50524389 bytes on disk I0108 18:49:54.024099 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8} I0108 18:49:54.024178 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 49454780 bytes on disk I0108 18:49:54.130270 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.106s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":105925,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":3} I0108 18:50:03.197029 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033351 I0108 18:50:06.344699 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033376 W0108 18:50:11.327620 24932 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.261s user 0.000s sys 0.000s I0108 18:50:11.328671 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 8.132s user 2.706s sys 0.165s Metrics: {"bytes_written":30680291,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":83083,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":245052,"lbm_writes_1-10_ms":39,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3024,"reinsert_count":0,"rows_written":1727701,"thread_start_us":38,"threads_started":1,"update_count":5993} I0108 18:50:11.328799 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 34398235 bytes on disk I0108 18:50:11.328917 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0108 18:50:11.328964 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.765915 I0108 18:50:13.647495 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033395 I0108 18:50:14.017314 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 7.672s user 2.566s sys 0.163s Metrics: {"bytes_written":31015173,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":2412,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":59130,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":178932,"lbm_writes_1-10_ms":39,"lbm_writes_lt_1ms":3057,"reinsert_count":0,"rows_written":1745712,"thread_start_us":31,"threads_started":1,"update_count":8024} I0108 18:50:14.021644 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 75795348 bytes of WAL I0108 18:50:14.035430 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 9 log segments from log reader I0108 18:50:14.035753 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000042 (ops 242845-248752) I0108 18:50:14.035849 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000043 (ops 248753-254637) I0108 18:50:14.035957 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000044 (ops 254638-260534) I0108 18:50:14.036326 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000045 (ops 260535-266469) I0108 18:50:14.036429 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000046 (ops 266470-272389) I0108 18:50:14.037459 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000047 (ops 272390-278340) I0108 18:50:14.037560 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000048 (ops 278341-284275) I0108 18:50:14.037621 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000049 (ops 284276-290244) I0108 18:50:14.042870 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000050 (ops 290245-296144) I0108 18:50:14.193857 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.158s user 0.005s sys 0.018s Metrics: {} I0108 18:50:14.195451 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.766404 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 300000 Mean: 2038.53 Percentiles: 0% (min) = 804 25% = 1492 50% (med) = 1692 75% = 1977 95% = 2814 99% = 8320 99.9% = 48704 99.99% = 86784 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:50:17.145995 24709 linked_list-test-util.h:519] Written 15000000 rows in chain I0108 18:50:26.178143 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 12.531s user 2.870s sys 0.226s Metrics: {"bytes_written":30835936,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":15321,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":30352,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":404665,"lbm_writes_1-10_ms":79,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":2992,"reinsert_count":0,"rows_written":1734773,"thread_start_us":96,"threads_started":1,"update_count":9014} I0108 18:50:26.178429 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 75533728 bytes of WAL I0108 18:50:26.178807 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 9 log segments from log reader I0108 18:50:26.178860 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000042 (ops 242991-248918) I0108 18:50:26.178946 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000043 (ops 248919-254887) I0108 18:50:26.179018 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000044 (ops 254888-260766) I0108 18:50:26.179085 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000045 (ops 260767-266656) I0108 18:50:26.179147 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000046 (ops 266657-272601) I0108 18:50:26.179242 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000047 (ops 272602-278449) I0108 18:50:26.179327 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000048 (ops 278450-284359) I0108 18:50:26.179385 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000049 (ops 284360-290245) I0108 18:50:26.179440 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000050 (ops 290246-296144) I0108 18:50:26.215672 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.037s user 0.000s sys 0.020s Metrics: {} I0108 18:50:26.215831 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 34579890 bytes on disk I0108 18:50:26.217532 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":154,"lbm_reads_lt_1ms":4} I0108 18:50:26.217711 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.768303 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 310000 Mean: 2047.65 Percentiles: 0% (min) = 804 25% = 1495 50% (med) = 1696 75% = 1984 95% = 2836 99% = 8424 99.9% = 49216 99.99% = 86784 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:50:41.565903 24709 linked_list-test-util.h:519] Written 15500000 rows in chain I0108 18:50:56.562685 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 45.234s user 10.647s sys 1.080s Metrics: {"bytes_written":140193738,"cfile_cache_hit":321,"cfile_cache_hit_bytes":43699387,"cfile_cache_miss":18325,"cfile_cache_miss_bytes":529087291,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":45440,"dirs.run_cpu_time_us":599,"dirs.run_wall_time_us":213123,"drs_written":5,"lbm_read_time_us":5138663,"lbm_reads_1-10_ms":352,"lbm_reads_10-100_ms":84,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":17965,"lbm_write_time_us":1455604,"lbm_writes_1-10_ms":316,"lbm_writes_10-100_ms":11,"lbm_writes_lt_1ms":13702,"num_input_rowsets":5,"rows_written":7912475,"spinlock_wait_cycles":530718080,"thread_start_us":760,"threads_started":13} I0108 18:50:56.562921 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 67704889 bytes of WAL I0108 18:50:56.563005 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 8 log segments from log reader I0108 18:50:56.563055 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000042 (ops 241868-247707) I0108 18:50:56.563132 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000043 (ops 247708-253894) I0108 18:50:56.563200 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000044 (ops 253895-259784) I0108 18:50:56.563266 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000045 (ops 259785-265702) I0108 18:50:56.563359 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000046 (ops 265703-271684) I0108 18:50:56.563417 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000047 (ops 271685-277548) I0108 18:50:56.563473 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000048 (ops 277549-283444) I0108 18:50:56.563544 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000049 (ops 283445-289369) I0108 18:50:56.608582 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.046s user 0.000s sys 0.021s Metrics: {} I0108 18:50:56.608798 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 168209886 bytes on disk I0108 18:50:56.911237 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.302s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":302070,"lbm_reads_1-10_ms":6,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":10} I0108 18:50:56.911383 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.315469 I0108 18:50:57.615911 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 43.420s user 10.429s sys 1.037s Metrics: {"bytes_written":140970605,"cfile_cache_hit":318,"cfile_cache_hit_bytes":43310721,"cfile_cache_miss":18434,"cfile_cache_miss_bytes":532732415,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":11989,"dirs.run_cpu_time_us":591,"dirs.run_wall_time_us":291029,"drs_written":5,"lbm_read_time_us":5877635,"lbm_reads_1-10_ms":311,"lbm_reads_10-100_ms":100,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":18100,"lbm_write_time_us":1529686,"lbm_writes_1-10_ms":319,"lbm_writes_10-100_ms":7,"lbm_writes_lt_1ms":13780,"num_input_rowsets":5,"rows_written":7956472,"spinlock_wait_cycles":123955456,"thread_start_us":571,"threads_started":13} I0108 18:50:57.616127 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.314354 I0108 18:50:57.792363 28737 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.66:58364 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=936507, attempt_no=0}) took 1156ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":2883,"cfile_cache_hit":1528,"cfile_cache_hit_bytes":6329876,"cfile_cache_miss":3669,"cfile_cache_miss_bytes":15206409,"lbm_read_time_us":934286,"lbm_reads_1-10_ms":35,"lbm_reads_10-100_ms":25,"lbm_reads_lt_1ms":3609,"num_ops":12743,"prepare.queue_time_us":17,"prepare.run_cpu_time_us":14765,"prepare.run_wall_time_us":39332,"raft.queue_time_us":25,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":135,"replication_time_us":2343,"spinlock_wait_cycles":76672}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 320000 Mean: 2061.85 Percentiles: 0% (min) = 804 25% = 1499 50% (med) = 1703 75% = 1995 95% = 2866 99% = 8536 99.9% = 49824 99.99% = 99136 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:51:07.898816 24709 linked_list-test-util.h:519] Written 16000000 rows in chain I0108 18:51:08.227623 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 42.009s user 10.498s sys 1.111s Metrics: {"bytes_written":142637998,"cfile_cache_hit":318,"cfile_cache_hit_bytes":43437489,"cfile_cache_miss":18654,"cfile_cache_miss_bytes":539740045,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":21871,"dirs.run_cpu_time_us":576,"dirs.run_wall_time_us":206524,"drs_written":5,"lbm_read_time_us":7247552,"lbm_reads_1-10_ms":290,"lbm_reads_10-100_ms":128,"lbm_reads_gt_100_ms":7,"lbm_reads_lt_1ms":18313,"lbm_write_time_us":1367055,"lbm_writes_1-10_ms":235,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14027,"num_input_rowsets":5,"rows_written":8051032,"spinlock_wait_cycles":320659968,"thread_start_us":961,"threads_started":13} I0108 18:51:08.231389 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.309984 W0108 18:51:19.517618 24807 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.267s user 0.000s sys 0.000s I0108 18:51:19.520668 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 21.904s user 5.301s sys 0.480s Metrics: {"bytes_written":73614077,"cfile_cache_hit":119,"cfile_cache_hit_bytes":15168894,"cfile_cache_miss":9675,"cfile_cache_miss_bytes":287500122,"cfile_init":14,"delta_iterators_relevant":6,"dirs.queue_time_us":9094,"dirs.run_cpu_time_us":354,"dirs.run_wall_time_us":160462,"drs_written":3,"lbm_read_time_us":6807645,"lbm_reads_1-10_ms":156,"lbm_reads_10-100_ms":147,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":9417,"lbm_write_time_us":941682,"lbm_writes_1-10_ms":102,"lbm_writes_10-100_ms":15,"lbm_writes_lt_1ms":7257,"num_input_rowsets":3,"rows_written":4155314,"spinlock_wait_cycles":78271872,"thread_start_us":203,"threads_started":3} I0108 18:51:19.520800 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 210326301 bytes on disk I0108 18:51:19.695353 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.174s user 0.001s sys 0.000s Metrics: {"cfile_init":6,"lbm_read_time_us":174119,"lbm_reads_1-10_ms":7,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":13} I0108 18:51:19.695557 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.041925 I0108 18:51:22.390151 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.694s user 0.747s sys 0.050s Metrics: {"bytes_written":9090392,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":124,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":72894,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869006,"lbm_writes_1-10_ms":23,"lbm_writes_10-100_ms":25,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":876,"reinsert_count":0,"rows_written":513104,"thread_start_us":29,"threads_started":1,"update_count":1287} I0108 18:51:22.390265 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41986530 bytes of WAL I0108 18:51:22.390396 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:51:22.390409 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000021 (ops 248752-261200) I0108 18:51:22.390439 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000022 (ops 261201-273645) I0108 18:51:22.390460 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000023 (ops 273646-286077) I0108 18:51:22.390481 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000024 (ops 286078-298493) I0108 18:51:22.390501 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000025 (ops 298494-310923) I0108 18:51:22.420307 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.030s user 0.002s sys 0.011s Metrics: {} I0108 18:51:22.420464 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.006841 I0108 18:51:30.360312 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 33.449s user 7.711s sys 0.768s Metrics: {"bytes_written":106517729,"cfile_cache_hit":140,"cfile_cache_hit_bytes":17356566,"cfile_cache_miss":14030,"cfile_cache_miss_bytes":421329522,"cfile_init":16,"delta_iterators_relevant":8,"dirs.queue_time_us":935,"dirs.run_cpu_time_us":433,"dirs.run_wall_time_us":296850,"drs_written":4,"lbm_read_time_us":9937864,"lbm_reads_1-10_ms":235,"lbm_reads_10-100_ms":212,"lbm_reads_gt_100_ms":14,"lbm_reads_lt_1ms":13633,"lbm_write_time_us":991640,"lbm_writes_1-10_ms":130,"lbm_writes_10-100_ms":12,"lbm_writes_lt_1ms":10521,"num_input_rowsets":4,"rows_written":6011884,"spinlock_wait_cycles":261003520,"thread_start_us":1445,"threads_started":12} I0108 18:51:30.360553 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 128233361 bytes on disk I0108 18:51:30.360713 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":27,"lbm_reads_lt_1ms":16} I0108 18:51:30.360803 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.045541 I0108 18:51:30.976454 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 22.745s user 5.539s sys 0.540s Metrics: {"bytes_written":75283353,"cfile_cache_hit":116,"cfile_cache_hit_bytes":14797910,"cfile_cache_miss":9900,"cfile_cache_miss_bytes":294934982,"cfile_init":14,"delta_iterators_relevant":6,"dirs.queue_time_us":5259,"dirs.run_cpu_time_us":401,"dirs.run_wall_time_us":115915,"drs_written":3,"lbm_read_time_us":6503682,"lbm_reads_1-10_ms":172,"lbm_reads_10-100_ms":152,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":9621,"lbm_write_time_us":1413973,"lbm_writes_1-10_ms":117,"lbm_writes_10-100_ms":33,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":7389,"num_input_rowsets":3,"rows_written":4249912,"spinlock_wait_cycles":62153216,"thread_start_us":577,"threads_started":11} I0108 18:51:30.976665 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 212487409 bytes on disk I0108 18:51:31.155951 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.179s user 0.000s sys 0.000s Metrics: {"cfile_init":6,"lbm_read_time_us":178937,"lbm_reads_1-10_ms":7,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":12} I0108 18:51:31.156111 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033432 I0108 18:51:31.965936 28990 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.66:58364 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=972388, attempt_no=0}) took 1490ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1518,"cfile_cache_hit":2244,"cfile_cache_hit_bytes":9552674,"cfile_cache_miss":2923,"cfile_cache_miss_bytes":12143162,"lbm_read_time_us":1331595,"lbm_reads_1-10_ms":23,"lbm_reads_10-100_ms":23,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":2873,"num_ops":12937,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":14158,"prepare.run_wall_time_us":63554,"raft.queue_time_us":79,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":119,"replication_time_us":4732,"spinlock_wait_cycles":87296}]]} I0108 18:51:33.357719 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.997s user 0.736s sys 0.055s Metrics: {"bytes_written":9751282,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":521,"dirs.run_cpu_time_us":62,"dirs.run_wall_time_us":63,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1470504,"lbm_writes_1-10_ms":23,"lbm_writes_10-100_ms":23,"lbm_writes_gt_100_ms":4,"lbm_writes_lt_1ms":939,"reinsert_count":0,"rows_written":549966,"thread_start_us":123,"threads_started":1,"update_count":3190} I0108 18:51:33.357898 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 42002334 bytes of WAL I0108 18:51:33.357971 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:51:33.358021 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000021 (ops 248008-260391) I0108 18:51:33.358114 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000022 (ops 260392-272849) I0108 18:51:33.358189 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000023 (ops 272850-285296) I0108 18:51:33.358286 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000024 (ops 285297-297667) I0108 18:51:33.358440 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000025 (ops 297668-310068) I0108 18:51:33.372011 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.014s user 0.000s sys 0.013s Metrics: {} I0108 18:51:33.372187 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 11017177 bytes on disk I0108 18:51:33.384793 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12417,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3} I0108 18:51:33.384878 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.006472 I0108 18:51:33.402940 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.247s user 0.571s sys 0.043s Metrics: {"bytes_written":7177942,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":4171,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":44621,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":864171,"lbm_writes_1-10_ms":17,"lbm_writes_10-100_ms":14,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":701,"reinsert_count":0,"rows_written":404647,"spinlock_wait_cycles":768,"thread_start_us":30,"threads_started":1,"update_count":42} I0108 18:51:33.403070 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 25181538 bytes of WAL I0108 18:51:33.403120 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 3 log segments from log reader I0108 18:51:33.403126 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000023 (ops 272596-284969) I0108 18:51:33.403148 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000024 (ops 284970-297345) I0108 18:51:33.403162 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000025 (ops 297346-309803) I0108 18:51:33.409068 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.006s user 0.000s sys 0.006s Metrics: {} I0108 18:51:33.409158 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 8053796 bytes on disk I0108 18:51:33.409273 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 18:51:33.409358 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.006434 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 330000 Mean: 2082.24 Percentiles: 0% (min) = 804 25% = 1502 50% (med) = 1707 75% = 2003 95% = 2884 99% = 8632 99.9% = 50752 99.99% = 124864 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:51:36.641894 24709 linked_list-test-util.h:519] Written 16500000 rows in chain I0108 18:51:37.046207 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 14.625s user 3.540s sys 0.339s Metrics: {"bytes_written":47763334,"cfile_cache_hit":115,"cfile_cache_hit_bytes":14633654,"cfile_cache_miss":6251,"cfile_cache_miss_bytes":181026414,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":6360,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":162726,"drs_written":2,"lbm_read_time_us":3371910,"lbm_reads_1-10_ms":104,"lbm_reads_10-100_ms":63,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":6132,"lbm_write_time_us":571656,"lbm_writes_1-10_ms":66,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":4715,"num_input_rowsets":3,"rows_written":2697964,"spinlock_wait_cycles":17132416,"thread_start_us":1308,"threads_started":10} I0108 18:51:37.046416 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.043703 I0108 18:51:39.873739 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.827s user 0.778s sys 0.058s Metrics: {"bytes_written":9437212,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":987,"dirs.run_cpu_time_us":54,"dirs.run_wall_time_us":224,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":395626,"lbm_writes_1-10_ms":30,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":924,"reinsert_count":0,"rows_written":533879,"thread_start_us":31,"threads_started":1,"update_count":47} I0108 18:51:39.874156 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41955614 bytes of WAL I0108 18:51:39.874240 24807 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:51:39.874294 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000022 (ops 260726-273181) I0108 18:51:39.874519 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000023 (ops 273182-285578) I0108 18:51:39.874598 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000024 (ops 285579-297963) I0108 18:51:39.874733 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000025 (ops 297964-310373) I0108 18:51:39.874804 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000026 (ops 310374-322842) I0108 18:51:39.901537 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.026s user 0.000s sys 0.014s Metrics: {} I0108 18:51:39.901713 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 57762295 bytes on disk I0108 18:51:39.946185 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.044s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":44183,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 18:51:39.946269 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.006291 I0108 18:51:47.238786 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 13.829s user 3.663s sys 0.381s Metrics: {"bytes_written":48398255,"cfile_cache_hit":101,"cfile_cache_hit_bytes":12523674,"cfile_cache_miss":6357,"cfile_cache_miss_bytes":186266994,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":1644,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":64626,"drs_written":2,"lbm_read_time_us":2023405,"lbm_reads_1-10_ms":119,"lbm_reads_10-100_ms":38,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":6246,"lbm_write_time_us":379877,"lbm_writes_1-10_ms":82,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":4769,"num_input_rowsets":3,"rows_written":2734046,"spinlock_wait_cycles":29486208,"thread_start_us":540,"threads_started":10} I0108 18:51:47.242437 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.034137 I0108 18:51:47.489502 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 14.105s user 3.654s sys 0.393s Metrics: {"bytes_written":48396027,"cfile_cache_hit":117,"cfile_cache_hit_bytes":14845699,"cfile_cache_miss":6327,"cfile_cache_miss_bytes":183426505,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":12663,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":32309,"drs_written":2,"lbm_read_time_us":2057370,"lbm_reads_1-10_ms":113,"lbm_reads_10-100_ms":39,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":6220,"lbm_write_time_us":620313,"lbm_writes_1-10_ms":102,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":4744,"num_input_rowsets":3,"rows_written":2733965,"spinlock_wait_cycles":27417344,"thread_start_us":533,"threads_started":10} I0108 18:51:47.493355 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 58607549 bytes on disk I0108 18:51:47.494480 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":966,"lbm_reads_lt_1ms":8} I0108 18:51:47.494544 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.046662 I0108 18:51:48.585986 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 1.343s user 0.544s sys 0.038s Metrics: {"bytes_written":7247888,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":84,"dirs.run_cpu_time_us":41,"dirs.run_wall_time_us":41,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":471254,"lbm_writes_1-10_ms":22,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":711,"reinsert_count":0,"rows_written":409295,"thread_start_us":63,"threads_started":1,"update_count":27} I0108 18:51:48.586169 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33608012 bytes of WAL I0108 18:51:48.586274 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:51:48.586319 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000023 (ops 273580-285991) I0108 18:51:48.586414 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000024 (ops 285992-298379) I0108 18:51:48.586490 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000025 (ops 298380-310893) I0108 18:51:48.586561 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000026 (ops 310894-323308) I0108 18:51:48.610669 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.024s user 0.001s sys 0.010s Metrics: {} I0108 18:51:48.611193 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.005889 I0108 18:51:50.224422 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.729s user 0.808s sys 0.050s Metrics: {"bytes_written":10068834,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":106,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":51943,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":857173,"lbm_writes_1-10_ms":26,"lbm_writes_10-100_ms":21,"lbm_writes_lt_1ms":979,"reinsert_count":0,"rows_written":570116,"thread_start_us":79,"threads_started":1,"update_count":161} I0108 18:51:50.229370 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 41994287 bytes of WAL I0108 18:51:50.230271 24932 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 5 log segments from log reader I0108 18:51:50.230356 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000022 (ops 260322-272690) I0108 18:51:50.230439 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000023 (ops 272691-285082) I0108 18:51:50.230504 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000024 (ops 285083-297509) I0108 18:51:50.230566 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000025 (ops 297510-309874) I0108 18:51:50.230623 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000026 (ops 309875-322283) I0108 18:51:50.268782 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.038s user 0.000s sys 0.014s Metrics: {} I0108 18:51:50.285790 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.005929 I0108 18:51:54.045740 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 14.099s user 3.609s sys 0.355s Metrics: {"bytes_written":48701272,"cfile_cache_hit":130,"cfile_cache_hit_bytes":16777893,"cfile_cache_miss":6364,"cfile_cache_miss_bytes":182840605,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":129851,"drs_written":2,"lbm_read_time_us":3117071,"lbm_reads_1-10_ms":108,"lbm_reads_10-100_ms":69,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":6236,"lbm_write_time_us":416306,"lbm_writes_1-10_ms":72,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":4804,"num_input_rowsets":3,"rows_written":2751326,"spinlock_wait_cycles":11008,"thread_start_us":508,"threads_started":10} I0108 18:51:54.045836 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 59002744 bytes on disk I0108 18:51:54.045930 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 340000 Mean: 2091.7 Percentiles: 0% (min) = 804 25% = 1504 50% (med) = 1710 75% = 2009 95% = 2898 99% = 8736 99.9% = 51360 99.99% = 126464 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:52:02.061367 24709 linked_list-test-util.h:519] Written 17000000 rows in chain I0108 18:52:02.803994 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 14.193s user 3.730s sys 0.406s Metrics: {"bytes_written":49307596,"cfile_cache_hit":87,"cfile_cache_hit_bytes":10276722,"cfile_cache_miss":6489,"cfile_cache_miss_bytes":192384088,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":9632,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":41670,"drs_written":2,"lbm_read_time_us":2288473,"lbm_reads_1-10_ms":120,"lbm_reads_10-100_ms":39,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":6379,"lbm_write_time_us":454632,"lbm_writes_1-10_ms":88,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":4853,"num_input_rowsets":3,"rows_written":2785996,"spinlock_wait_cycles":40312704,"thread_start_us":2716,"threads_started":10} I0108 18:52:02.804177 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 59787004 bytes on disk I0108 18:52:02.808821 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8} I0108 18:52:02.808957 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 58608479 bytes on disk I0108 18:52:02.909029 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.100s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":99774,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":3} I0108 18:52:03.654685 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 13.369s user 3.634s sys 0.433s Metrics: {"bytes_written":49310749,"cfile_cache_hit":106,"cfile_cache_hit_bytes":13041411,"cfile_cache_miss":6470,"cfile_cache_miss_bytes":189060883,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":8354,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":88008,"drs_written":2,"lbm_read_time_us":2153683,"lbm_reads_1-10_ms":92,"lbm_reads_10-100_ms":38,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":6389,"lbm_write_time_us":555521,"lbm_writes_1-10_ms":81,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":4860,"num_input_rowsets":3,"rows_written":2786130,"spinlock_wait_cycles":45157888,"thread_start_us":354,"threads_started":10} I0108 18:52:03.654860 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 59791489 bytes on disk I0108 18:52:03.663532 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":144,"lbm_reads_lt_1ms":8} I0108 18:52:11.419536 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033359 I0108 18:52:14.073801 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033349 I0108 18:52:18.533495 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 7.114s user 2.553s sys 0.201s Metrics: {"bytes_written":29299998,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":11309,"dirs.run_cpu_time_us":118,"dirs.run_wall_time_us":107675,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":213945,"lbm_writes_1-10_ms":35,"lbm_writes_lt_1ms":2903,"reinsert_count":0,"rows_written":1656057,"thread_start_us":32,"threads_started":1,"update_count":13122} I0108 18:52:18.533668 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 75740047 bytes of WAL I0108 18:52:18.533746 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 9 log segments from log reader I0108 18:52:18.533790 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000050 (ops 289370-295316) I0108 18:52:18.533883 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000051 (ops 295317-301266) I0108 18:52:18.533918 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000052 (ops 301267-307182) I0108 18:52:18.533941 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000053 (ops 307183-313081) I0108 18:52:18.533993 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000054 (ops 313082-319045) I0108 18:52:18.534168 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000055 (ops 319046-324918) I0108 18:52:18.534229 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000056 (ops 324919-330858) I0108 18:52:18.534314 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000057 (ops 330859-336690) I0108 18:52:18.534379 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000058 (ops 336691-342588) I0108 18:52:18.586448 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.053s user 0.000s sys 0.029s Metrics: {} I0108 18:52:18.586596 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.615434 I0108 18:52:21.245316 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 7.171s user 2.524s sys 0.161s Metrics: {"bytes_written":29128602,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":5927,"dirs.run_cpu_time_us":69,"dirs.run_wall_time_us":68,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":388038,"lbm_writes_1-10_ms":47,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":2879,"reinsert_count":0,"rows_written":1649752,"thread_start_us":31,"threads_started":1,"update_count":6813} I0108 18:52:21.246680 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 67604921 bytes of WAL I0108 18:52:21.246907 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 8 log segments from log reader I0108 18:52:21.247012 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000051 (ops 296145-302288) I0108 18:52:21.247262 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000052 (ops 302289-308284) I0108 18:52:21.247350 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000053 (ops 308285-314205) I0108 18:52:21.247438 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000054 (ops 314206-320126) I0108 18:52:21.247526 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000055 (ops 320127-326012) I0108 18:52:21.247586 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000056 (ops 326013-331991) I0108 18:52:21.247639 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000057 (ops 331992-337944) I0108 18:52:21.247694 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000058 (ops 337945-343808) I0108 18:52:21.317026 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.070s user 0.000s sys 0.024s Metrics: {} I0108 18:52:21.317181 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 32872115 bytes on disk I0108 18:52:21.317328 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 18:52:21.317425 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.616708 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 350000 Mean: 2094.23 Percentiles: 0% (min) = 804 25% = 1505 50% (med) = 1711 75% = 2010 95% = 2900 99% = 8768 99.9% = 51200 99.99% = 124864 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:52:25.263489 24709 linked_list-test-util.h:519] Written 17500000 rows in chain I0108 18:52:26.180728 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033334 W0108 18:52:36.424731 25056 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.323s user 0.000s sys 0.000s I0108 18:52:36.425640 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 10.245s user 2.750s sys 0.174s Metrics: {"bytes_written":30463950,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":103,"dirs.run_wall_time_us":345241,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":277251,"lbm_writes_1-10_ms":56,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":3005,"reinsert_count":0,"rows_written":1727664,"thread_start_us":144,"threads_started":1,"update_count":4778} I0108 18:52:36.425861 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 34481563 bytes on disk I0108 18:52:36.426522 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 18:52:36.426638 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.619394 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 360000 Mean: 2103.47 Percentiles: 0% (min) = 804 25% = 1508 50% (med) = 1716 75% = 2019 95% = 2918 99% = 8824 99.9% = 51392 99.99% = 128768 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:52:50.882370 24709 linked_list-test-util.h:519] Written 18000000 rows in chain W0108 18:52:58.840776 24932 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.210s user 0.001s sys 0.000s I0108 18:52:58.843268 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 40.257s user 10.241s sys 1.089s Metrics: {"bytes_written":135614687,"cfile_cache_hit":269,"cfile_cache_hit_bytes":35208224,"cfile_cache_miss":17805,"cfile_cache_miss_bytes":521807048,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":25669,"dirs.run_cpu_time_us":587,"dirs.run_wall_time_us":256105,"drs_written":5,"lbm_read_time_us":4844612,"lbm_reads_1-10_ms":301,"lbm_reads_10-100_ms":84,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":17499,"lbm_write_time_us":1579356,"lbm_writes_1-10_ms":263,"lbm_writes_10-100_ms":13,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":13320,"num_input_rowsets":5,"rows_written":7667999,"spinlock_wait_cycles":430772608,"thread_start_us":125,"threads_started":4} I0108 18:52:58.843417 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.523244 I0108 18:52:59.447293 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 38.130s user 10.134s sys 0.991s Metrics: {"bytes_written":136280446,"cfile_cache_hit":268,"cfile_cache_hit_bytes":35254015,"cfile_cache_miss":17894,"cfile_cache_miss_bytes":524604695,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":14318,"dirs.run_cpu_time_us":507,"dirs.run_wall_time_us":218221,"drs_written":5,"lbm_read_time_us":5230105,"lbm_reads_1-10_ms":335,"lbm_reads_10-100_ms":70,"lbm_reads_gt_100_ms":12,"lbm_reads_lt_1ms":17561,"lbm_write_time_us":1445239,"lbm_writes_1-10_ms":248,"lbm_writes_10-100_ms":12,"lbm_writes_lt_1ms":13406,"num_input_rowsets":5,"rows_written":7705607,"spinlock_wait_cycles":236510080,"thread_start_us":164,"threads_started":4} I0108 18:52:59.447439 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 164919428 bytes on disk I0108 18:52:59.447574 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":20,"lbm_reads_lt_1ms":20} I0108 18:52:59.447629 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.521292 I0108 18:53:01.073572 29439 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.66:58364 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=1076302, attempt_no=0}) took 1693ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":9,"cfile_cache_hit":1657,"cfile_cache_hit_bytes":7150667,"cfile_cache_miss":2148,"cfile_cache_miss_bytes":8960301,"lbm_read_time_us":1652029,"lbm_reads_1-10_ms":59,"lbm_reads_10-100_ms":28,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":2058,"num_ops":4966,"prepare.queue_time_us":27,"prepare.run_cpu_time_us":4628,"prepare.run_wall_time_us":11869,"raft.queue_time_us":81,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":54,"replication_time_us":1127,"spinlock_wait_cycles":384}]]} I0108 18:53:18.705662 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 42.279s user 10.371s sys 1.009s Metrics: {"bytes_written":139307022,"cfile_cache_hit":246,"cfile_cache_hit_bytes":32238752,"cfile_cache_miss":18318,"cfile_cache_miss_bytes":540241676,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":24174,"dirs.run_cpu_time_us":574,"dirs.run_wall_time_us":447589,"drs_written":5,"lbm_read_time_us":8368407,"lbm_reads_1-10_ms":317,"lbm_reads_10-100_ms":133,"lbm_reads_gt_100_ms":19,"lbm_reads_lt_1ms":17933,"lbm_write_time_us":1243690,"lbm_writes_1-10_ms":237,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":13726,"num_input_rowsets":5,"rows_written":7878151,"spinlock_wait_cycles":202401408,"thread_start_us":1473,"threads_started":13} I0108 18:53:18.705821 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 76200616 bytes of WAL I0108 18:53:18.705878 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 9 log segments from log reader I0108 18:53:18.705891 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000051 (ops 296145-302210) I0108 18:53:18.705929 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000052 (ops 302211-308285) I0108 18:53:18.705951 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000053 (ops 308286-314210) I0108 18:53:18.705972 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000054 (ops 314211-320226) I0108 18:53:18.705991 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000055 (ops 320227-326101) I0108 18:53:18.706009 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000056 (ops 326102-332047) I0108 18:53:18.706032 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000057 (ops 332048-337958) I0108 18:53:18.706074 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000058 (ops 337959-343953) I0108 18:53:18.706092 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000059 (ops 343954-349772) I0108 18:53:18.753032 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.047s user 0.000s sys 0.023s Metrics: {} I0108 18:53:18.753314 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.351488 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 370000 Mean: 2124.76 Percentiles: 0% (min) = 804 25% = 1511 50% (med) = 1721 75% = 2028 95% = 2942 99% = 8936 99.9% = 52128 99.99% = 156672 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:53:21.067364 24709 linked_list-test-util.h:519] Written 18500000 rows in chain I0108 18:53:41.535008 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 42.691s user 9.980s sys 1.018s Metrics: {"bytes_written":134496055,"cfile_cache_hit":164,"cfile_cache_hit_bytes":20679018,"cfile_cache_miss":17742,"cfile_cache_miss_bytes":535556676,"cfile_init":18,"delta_iterators_relevant":8,"dirs.queue_time_us":26154,"dirs.run_cpu_time_us":478,"dirs.run_wall_time_us":124958,"drs_written":4,"lbm_read_time_us":9636817,"lbm_reads_1-10_ms":330,"lbm_reads_10-100_ms":198,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":17278,"lbm_write_time_us":2554298,"lbm_writes_1-10_ms":247,"lbm_writes_10-100_ms":33,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":13180,"num_input_rowsets":4,"rows_written":7602283,"spinlock_wait_cycles":348353152,"thread_start_us":381,"threads_started":12} I0108 18:53:41.535204 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 246886739 bytes on disk I0108 18:53:41.735687 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.200s user 0.001s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":200152,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":21} I0108 18:53:41.735824 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.035661 I0108 18:53:42.093255 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 42.646s user 9.767s sys 0.996s Metrics: {"bytes_written":134500325,"cfile_cache_hit":188,"cfile_cache_hit_bytes":24260909,"cfile_cache_miss":17724,"cfile_cache_miss_bytes":532082373,"cfile_init":17,"delta_iterators_relevant":8,"dirs.queue_time_us":20868,"dirs.run_cpu_time_us":591,"dirs.run_wall_time_us":332498,"drs_written":5,"lbm_read_time_us":9805722,"lbm_reads_1-10_ms":303,"lbm_reads_10-100_ms":213,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":17265,"lbm_write_time_us":2919976,"lbm_writes_1-10_ms":236,"lbm_writes_10-100_ms":43,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":13208,"num_input_rowsets":4,"rows_written":7602334,"spinlock_wait_cycles":151526528,"thread_start_us":656,"threads_started":13} I0108 18:53:42.093430 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 163695840 bytes on disk I0108 18:53:42.093559 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":32,"lbm_reads_lt_1ms":20} I0108 18:53:42.093601 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.038807 I0108 18:53:44.510897 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.417s user 0.479s sys 0.021s Metrics: {"bytes_written":7987206,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":75,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":421569,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1861487,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":47,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":768,"reinsert_count":0,"rows_written":453905,"thread_start_us":35,"threads_started":1,"update_count":9} I0108 18:53:44.511047 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33584741 bytes of WAL I0108 18:53:44.511139 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:53:44.511152 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000026 (ops 310924-323397) I0108 18:53:44.511186 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000027 (ops 323398-335805) I0108 18:53:44.511204 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000028 (ops 335806-348259) I0108 18:53:44.511245 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000029 (ops 348260-360722) I0108 18:53:44.522385 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.011s user 0.000s sys 0.011s Metrics: {} I0108 18:53:44.522586 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 9065448 bytes on disk I0108 18:53:44.529842 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":6966,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0108 18:53:44.529990 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.002060 I0108 18:53:44.884500 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 3.149s user 0.488s sys 0.022s Metrics: {"bytes_written":7376852,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":63424,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2544236,"lbm_writes_1-10_ms":10,"lbm_writes_10-100_ms":70,"lbm_writes_lt_1ms":677,"reinsert_count":0,"rows_written":417465,"thread_start_us":41,"threads_started":1,"update_count":1874} I0108 18:53:44.884619 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 33565389 bytes of WAL I0108 18:53:44.884672 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 4 log segments from log reader I0108 18:53:44.884685 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000026 (ops 310069-322461) I0108 18:53:44.884713 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000027 (ops 322462-334886) I0108 18:53:44.884737 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000028 (ops 334887-347311) I0108 18:53:44.884755 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000029 (ops 347312-359696) I0108 18:53:44.892664 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.008s user 0.000s sys 0.008s Metrics: {} I0108 18:53:44.892755 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.002047 I0108 18:53:45.544266 29710 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.24.33.66:58364 (ReqId={client: 20060d3408dd4d3a8f0e19885a75dabd, seq_no=1121020, attempt_no=0}) took 3604ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":254,"cfile_cache_hit":1591,"cfile_cache_hit_bytes":6598570,"cfile_cache_miss":2832,"cfile_cache_miss_bytes":11628192,"lbm_read_time_us":3539326,"lbm_reads_1-10_ms":26,"lbm_reads_10-100_ms":103,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":2701,"num_ops":7183,"prepare.queue_time_us":14,"prepare.run_cpu_time_us":5300,"prepare.run_wall_time_us":5300,"raft.queue_time_us":83,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"replication_time_us":1768,"spinlock_wait_cycles":896}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 380000 Mean: 2141.88 Percentiles: 0% (min) = 804 25% = 1513 50% (med) = 1724 75% = 2034 95% = 2954 99% = 9000 99.9% = 52864 99.99% = 175488 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:53:50.244247 24709 linked_list-test-util.h:519] Written 19000000 rows in chain I0108 18:53:59.802345 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 41.049s user 9.639s sys 0.975s Metrics: {"bytes_written":134477231,"cfile_cache_hit":190,"cfile_cache_hit_bytes":24332473,"cfile_cache_miss":17722,"cfile_cache_miss_bytes":532146709,"cfile_init":18,"delta_iterators_relevant":8,"dirs.queue_time_us":19804,"dirs.run_cpu_time_us":508,"dirs.run_wall_time_us":203531,"drs_written":4,"lbm_read_time_us":11634642,"lbm_reads_1-10_ms":295,"lbm_reads_10-100_ms":288,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":17200,"lbm_write_time_us":1622188,"lbm_writes_1-10_ms":192,"lbm_writes_10-100_ms":17,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":13251,"num_input_rowsets":4,"rows_written":7602346,"spinlock_wait_cycles":310430848,"thread_start_us":795,"threads_started":12} I0108 18:53:59.802546 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.243187 I0108 18:54:01.031045 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 16.138s user 3.912s sys 0.370s Metrics: {"bytes_written":55714401,"cfile_cache_hit":103,"cfile_cache_hit_bytes":12770577,"cfile_cache_miss":7331,"cfile_cache_miss_bytes":217361911,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":7165,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":182235,"drs_written":2,"lbm_read_time_us":4857684,"lbm_reads_1-10_ms":78,"lbm_reads_10-100_ms":88,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":7207,"lbm_write_time_us":672854,"lbm_writes_1-10_ms":90,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":5489,"num_input_rowsets":3,"rows_written":3151430,"spinlock_wait_cycles":97643904,"thread_start_us":743,"threads_started":10} I0108 18:54:01.031183 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.036335 I0108 18:54:03.520565 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 2.489s user 0.562s sys 0.034s Metrics: {"bytes_written":7248935,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":122,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":65302,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1263326,"lbm_writes_1-10_ms":23,"lbm_writes_10-100_ms":28,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":693,"reinsert_count":0,"rows_written":411538,"spinlock_wait_cycles":896,"thread_start_us":82,"threads_started":1,"update_count":51} I0108 18:54:03.520754 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33594530 bytes of WAL I0108 18:54:03.520826 24932 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:54:03.520869 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000027 (ops 322284-334703) I0108 18:54:03.520938 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000028 (ops 334704-347059) I0108 18:54:03.522163 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000029 (ops 347060-359509) I0108 18:54:03.522236 24932 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000030 (ops 359510-371951) I0108 18:54:03.564545 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.044s user 0.000s sys 0.012s Metrics: {} I0108 18:54:03.564760 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 68060331 bytes on disk I0108 18:54:03.615391 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.051s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":50444,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":4} I0108 18:54:03.615543 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.001860 I0108 18:54:04.076205 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 19.544s user 3.971s sys 0.360s Metrics: {"bytes_written":55717037,"cfile_cache_hit":110,"cfile_cache_hit_bytes":13570587,"cfile_cache_miss":7326,"cfile_cache_miss_bytes":216377961,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":96655,"drs_written":2,"lbm_read_time_us":5564227,"lbm_reads_1-10_ms":98,"lbm_reads_10-100_ms":96,"lbm_reads_gt_100_ms":14,"lbm_reads_lt_1ms":7166,"lbm_write_time_us":3159924,"lbm_writes_1-10_ms":82,"lbm_writes_10-100_ms":61,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":5443,"num_input_rowsets":3,"rows_written":3151869,"spinlock_wait_cycles":106593024,"thread_start_us":4090,"threads_started":10} I0108 18:54:04.076316 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 68074874 bytes on disk I0108 18:54:04.076514 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":90,"lbm_reads_lt_1ms":8} I0108 18:54:04.076611 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.040056 I0108 18:54:06.909418 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 2.833s user 0.670s sys 0.056s Metrics: {"bytes_written":7952027,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":768,"dirs.run_cpu_time_us":47,"dirs.run_wall_time_us":47,"drs_written":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4,"lbm_write_time_us":460230,"lbm_writes_1-10_ms":30,"lbm_writes_10-100_ms":14,"lbm_writes_lt_1ms":771,"reinsert_count":0,"rows_written":451025,"thread_start_us":33,"threads_started":1,"update_count":1347} I0108 18:54:06.910305 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 33611757 bytes of WAL I0108 18:54:06.910567 24807 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 4 log segments from log reader I0108 18:54:06.911079 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000027 (ops 322843-335227) I0108 18:54:06.911170 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000028 (ops 335228-347672) I0108 18:54:06.911264 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000029 (ops 347673-360071) I0108 18:54:06.911342 24807 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000030 (ops 360072-372568) I0108 18:54:06.960850 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.050s user 0.000s sys 0.013s Metrics: {} I0108 18:54:06.961083 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.001523 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 390000 Mean: 2156.36 Percentiles: 0% (min) = 804 25% = 1515 50% (med) = 1727 75% = 2041 95% = 2970 99% = 9072 99.9% = 53312 99.99% = 191872 100% (max) = 2453906 *NOTE: some values were greater than highest trackable value I0108 18:54:18.553583 24709 linked_list-test-util.h:519] Written 19500000 rows in chain I0108 18:54:20.838618 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 17.223s user 4.255s sys 0.483s Metrics: {"bytes_written":56518119,"cfile_cache_hit":101,"cfile_cache_hit_bytes":12454653,"cfile_cache_miss":7445,"cfile_cache_miss_bytes":221194681,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":8406,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":137630,"drs_written":2,"lbm_read_time_us":2488574,"lbm_reads_1-10_ms":153,"lbm_reads_10-100_ms":43,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":7298,"lbm_write_time_us":721914,"lbm_writes_1-10_ms":85,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":5574,"num_input_rowsets":3,"rows_written":3197668,"spinlock_wait_cycles":78094976,"thread_start_us":496,"threads_started":10} I0108 18:54:20.838788 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033974 I0108 18:54:21.135989 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 21.333s user 4.751s sys 0.504s Metrics: {"bytes_written":67249588,"cfile_cache_hit":54,"cfile_cache_hit_bytes":6005337,"cfile_cache_miss":8902,"cfile_cache_miss_bytes":273990269,"cfile_init":11,"delta_iterators_relevant":4,"dirs.queue_time_us":1080,"dirs.run_cpu_time_us":384,"dirs.run_wall_time_us":197089,"drs_written":3,"lbm_read_time_us":4505014,"lbm_reads_1-10_ms":183,"lbm_reads_10-100_ms":79,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":8674,"lbm_write_time_us":1811022,"lbm_writes_1-10_ms":119,"lbm_writes_10-100_ms":38,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":6600,"mutex_wait_us":1,"num_input_rowsets":2,"rows_written":3801201,"spinlock_wait_cycles":93826432,"thread_start_us":482,"threads_started":10} I0108 18:54:21.136166 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 251697320 bytes on disk I0108 18:54:21.350000 24709 linked_list-test-util.h:528] Sample at HT timestamp: 6465693536310034432 Inserted count: 19548500 I0108 18:54:21.350028 24709 linked_list-test-util.h:532] Finished inserting list. Added 19548500 in chain I0108 18:54:21.350033 24709 linked_list-test-util.h:533] Last entries inserted had keys: I0108 18:54:21.350037 24709 linked_list-test-util.h:535] 0: -9119375304437334016 I0108 18:54:21.350041 24709 linked_list-test-util.h:535] 1: 6454049633963606017 I0108 18:54:21.350044 24709 linked_list-test-util.h:535] 2: 3581011973631705090 I0108 18:54:21.350049 24709 linked_list-test-util.h:535] 3: 707974313299804163 I0108 18:54:21.350052 24709 linked_list-test-util.h:535] 4: 7162164684751831044 I0108 18:54:21.350055 24709 linked_list-test-util.h:535] 5: 4288845549443219461 I0108 18:54:21.350059 24709 linked_list-test-util.h:535] 6: 1415948626599608326 I0108 18:54:21.350061 24709 linked_list-test-util.h:535] 7: -1457089033732292601 I0108 18:54:21.350065 24709 linked_list-test-util.h:535] 8: 4997101337719734280 I0108 18:54:21.350069 24709 linked_list-test-util.h:535] 9: 2123782202411122697 I0108 18:54:21.350071 24709 linked_list-test-util.h:535] 10: -749255457920778230 I0108 18:54:21.350075 24709 linked_list-test-util.h:535] 11: -3622293118252679157 I0108 18:54:21.350078 24709 linked_list-test-util.h:535] 12: 2832037990687637516 I0108 18:54:21.350081 24709 linked_list-test-util.h:535] 13: -41281144620974067 I0108 18:54:21.350085 24709 linked_list-test-util.h:535] 14: -2914318804952874994 I0108 18:54:21.350088 24709 linked_list-test-util.h:535] 15: -5787356465284775921 I0108 18:54:21.350091 24709 linked_list-test-util.h:535] 16: 666833906167250960 I0108 18:54:21.350095 24709 linked_list-test-util.h:535] 17: -2206344491653070831 I0108 18:54:21.350098 24709 linked_list-test-util.h:535] 18: -5079382151984971758 I0108 18:54:21.350101 24709 linked_list-test-util.h:535] 19: -7952419812316872685 I0108 18:54:21.350105 24709 linked_list-test-util.h:535] 20: -1498229440864845804 I0108 18:54:21.350108 24709 linked_list-test-util.h:535] 21: -4371548576173457387 I0108 18:54:21.350111 24709 linked_list-test-util.h:535] 22: -7244586236505358314 I0108 18:54:21.350114 24709 linked_list-test-util.h:535] 23: 8329260914360582167 I0108 18:54:21.350118 24709 linked_list-test-util.h:535] 24: -3663292787896942568 I0108 18:54:21.350121 24709 linked_list-test-util.h:535] 25: -6536611923205554151 I0108 18:54:21.350124 24709 linked_list-test-util.h:535] 26: 9037094490172096538 I0108 18:54:21.350128 24709 linked_list-test-util.h:535] 27: -2955459212085428197 I0108 18:54:21.350131 24709 linked_list-test-util.h:535] 28: -5828496872417329124 I0108 18:54:21.350134 24709 linked_list-test-util.h:535] 29: -8701675270237650915 I0108 18:54:21.350137 24709 linked_list-test-util.h:535] 30: 6872031143139999774 I0108 18:54:21.350142 24709 linked_list-test-util.h:535] 31: -5120522559117524961 I0108 18:54:21.350144 24709 linked_list-test-util.h:535] 32: -7993560219449425888 I0108 18:54:21.350147 24709 linked_list-test-util.h:535] 33: 7580146193928224801 I0108 18:54:21.350152 24709 linked_list-test-util.h:535] 34: 4706827058619613218 I0108 18:54:21.350154 24709 linked_list-test-util.h:535] 35: -7285585906149621725 I0108 18:54:21.350158 24709 linked_list-test-util.h:535] 36: 8288120507228028964 I0108 18:54:21.350167 24709 linked_list-test-util.h:535] 37: 5415082846896128037 I0108 18:54:21.350170 24709 linked_list-test-util.h:535] 38: 2541763711587516454 I0108 18:54:21.350174 24709 linked_list-test-util.h:535] 39: 8995954083039543335 I0108 18:54:21.350178 24709 linked_list-test-util.h:535] 40: 6123057160195932200 I0108 18:54:21.350180 24709 linked_list-test-util.h:535] 41: 3250019499864031273 I0108 18:54:21.350184 24709 linked_list-test-util.h:535] 42: 376700364555419690 I0108 18:54:21.350188 24709 linked_list-test-util.h:535] 43: 6830890736007446571 I0108 18:54:21.350190 24709 linked_list-test-util.h:535] 44: 3957853075675545644 I0108 18:54:21.350194 24709 linked_list-test-util.h:535] 45: 1084815415343644717 I0108 18:54:21.350198 24709 linked_list-test-util.h:535] 46: -1788362982476677074 I0108 18:54:21.350201 24709 linked_list-test-util.h:535] 47: 4665827388975349807 I0108 18:54:21.350204 24709 linked_list-test-util.h:535] 48: 1792789728643448880 I0108 18:54:21.350208 24709 linked_list-test-util.h:535] 49: -1080247931688452047 I0108 18:54:21.484889 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.349s user 0.001s sys 0.000s Metrics: {"cfile_init":8,"lbm_read_time_us":348239,"lbm_reads_1-10_ms":9,"lbm_reads_10-100_ms":7,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":15} I0108 18:54:21.485021 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.046689 I0108 18:54:21.575430 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578537861018563 usec, L: 1 Expected Rows: 7332500 I0108 18:54:21.772236 24709 linked_list-test-util.h:639] Calling callback on tserver 53135810f2d84759ac110009de750c3d I0108 18:54:22.214715 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 15.253s user 4.205s sys 0.397s Metrics: {"bytes_written":56603418,"cfile_cache_hit":118,"cfile_cache_hit_bytes":15037302,"cfile_cache_miss":7430,"cfile_cache_miss_bytes":218805644,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":222,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":96078,"drs_written":2,"lbm_read_time_us":2770474,"lbm_reads_1-10_ms":123,"lbm_reads_10-100_ms":39,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":7314,"lbm_write_time_us":765214,"lbm_writes_1-10_ms":91,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":5581,"num_input_rowsets":3,"rows_written":3202351,"spinlock_wait_cycles":72475904,"thread_start_us":428,"threads_started":10} I0108 18:54:22.214927 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033603 I0108 18:54:22.473049 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.988s user 0.718s sys 0.030s Metrics: {"bytes_written":9274156,"cfile_init":1,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":73063,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":26524,"lbm_writes_lt_1ms":939,"rows_written":523610,"thread_start_us":81,"threads_started":2,"wal-append.queue_time_us":12381} I0108 18:54:22.473378 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 41969798 bytes of WAL I0108 18:54:22.473938 25056 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 5 log segments from log reader I0108 18:54:22.473990 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000026 (ops 309804-322194) I0108 18:54:22.474118 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000027 (ops 322195-334616) I0108 18:54:22.474200 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000028 (ops 334617-346984) I0108 18:54:22.474335 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000029 (ops 346985-359403) I0108 18:54:22.474413 25056 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000030 (ops 359404-371825) I0108 18:54:22.501693 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.028s user 0.001s sys 0.013s Metrics: {} I0108 18:54:22.501993 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.000951 I0108 18:54:24.650279 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 2.435s user 1.813s sys 0.087s Metrics: {"bytes_written":26991509,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":69699,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":237259,"lbm_writes_10-100_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":2704,"rows_written":1525929,"thread_start_us":61,"threads_started":2,"wal-append.queue_time_us":53} I0108 18:54:24.650444 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 67163569 bytes of WAL I0108 18:54:24.650511 24807 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 8 log segments from log reader I0108 18:54:24.650526 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000059 (ops 343809-349697) I0108 18:54:24.650578 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000060 (ops 349698-355648) I0108 18:54:24.650647 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000061 (ops 355649-361522) I0108 18:54:24.650719 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000062 (ops 361523-367363) I0108 18:54:24.650749 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000063 (ops 367364-373372) I0108 18:54:24.650768 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000064 (ops 373373-379244) I0108 18:54:24.650789 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000065 (ops 379245-385183) I0108 18:54:24.650816 24807 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000066 (ops 385184-391052) I0108 18:54:24.665661 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.015s user 0.001s sys 0.014s Metrics: {} I0108 18:54:24.665786 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.518454 I0108 18:54:26.121915 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 5.283s user 2.195s sys 0.146s Metrics: {"bytes_written":27775474,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":119,"dirs.run_wall_time_us":44352,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1857322,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":29,"lbm_writes_gt_100_ms":6,"lbm_writes_lt_1ms":2719,"reinsert_count":0,"rows_written":1563510,"thread_start_us":63,"threads_started":2,"update_count":11708,"wal-append.queue_time_us":75} I0108 18:54:26.122081 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 67314036 bytes of WAL I0108 18:54:26.122159 24932 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 8 log segments from log reader I0108 18:54:26.122228 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000059 (ops 342589-348527) I0108 18:54:26.122360 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000060 (ops 348528-354492) I0108 18:54:26.122431 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000061 (ops 354493-360307) I0108 18:54:26.122495 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000062 (ops 360308-366232) I0108 18:54:26.122560 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000063 (ops 366233-372148) I0108 18:54:26.122623 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000064 (ops 372149-378001) I0108 18:54:26.122648 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000065 (ops 378002-383940) I0108 18:54:26.122666 24932 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000066 (ops 383941-389873) I0108 18:54:26.140664 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.018s user 0.000s sys 0.018s Metrics: {} I0108 18:54:26.140789 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 69126046 bytes on disk I0108 18:54:26.212482 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.072s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":71501,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 18:54:26.212613 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 31099591 bytes on disk I0108 18:54:26.212721 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 18:54:26.212810 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.508226 I0108 18:54:34.805891 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 12.303s user 3.541s sys 0.293s Metrics: {"bytes_written":57886088,"cfile_cache_hit":111,"cfile_cache_hit_bytes":14015385,"cfile_cache_miss":7567,"cfile_cache_miss_bytes":223787515,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":143771,"drs_written":2,"lbm_read_time_us":6960279,"lbm_reads_1-10_ms":92,"lbm_reads_10-100_ms":178,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":7336,"lbm_write_time_us":176046,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":5772,"num_input_rowsets":3,"rows_written":3257656,"thread_start_us":491,"threads_started":11,"wal-append.queue_time_us":85} I0108 18:54:34.806107 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 70471595 bytes on disk I0108 18:54:34.997542 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.191s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":191213,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":3} I0108 18:54:34.997678 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.046225 I0108 18:54:35.775732 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.778s user 0.568s sys 0.026s Metrics: {"bytes_written":8378523,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":108,"dirs.run_wall_time_us":79875,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":19155,"lbm_writes_lt_1ms":850,"rows_written":472138,"thread_start_us":60,"threads_started":2,"wal-append.queue_time_us":57} I0108 18:54:35.775825 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 33568440 bytes of WAL I0108 18:54:35.775866 25056 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 4 log segments from log reader I0108 18:54:35.775880 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000027 (ops 323309-335754) I0108 18:54:35.775904 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000028 (ops 335755-348160) I0108 18:54:35.775918 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000029 (ops 348161-360557) I0108 18:54:35.775928 25056 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000030 (ops 360558-373026) I0108 18:54:35.783663 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.008s user 0.000s sys 0.008s Metrics: {} I0108 18:54:35.783766 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 9413105 bytes on disk I0108 18:54:35.783862 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 18:54:35.783910 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.001227 I0108 18:54:50.448807 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 14.665s user 3.334s sys 0.249s Metrics: {"bytes_written":57886588,"cfile_cache_hit":111,"cfile_cache_hit_bytes":14005983,"cfile_cache_miss":7573,"cfile_cache_miss_bytes":224047709,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":183,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":193444,"drs_written":2,"lbm_read_time_us":10347300,"lbm_reads_1-10_ms":86,"lbm_reads_10-100_ms":224,"lbm_reads_gt_100_ms":13,"lbm_reads_lt_1ms":7294,"lbm_write_time_us":120383,"lbm_writes_lt_1ms":5775,"num_input_rowsets":3,"rows_written":3258134,"thread_start_us":487,"threads_started":11,"wal-append.queue_time_us":56} I0108 18:54:50.449030 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 70493704 bytes on disk I0108 18:54:50.449167 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 18:54:50.449244 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.037229 I0108 18:54:52.960762 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 2.511s user 1.670s sys 0.088s Metrics: {"bytes_written":23979487,"cfile_init":1,"dirs.queue_time_us":14939,"dirs.run_cpu_time_us":98,"dirs.run_wall_time_us":120989,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":113344,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2398,"rows_written":1353447,"thread_start_us":66,"threads_started":2,"wal-append.queue_time_us":74} I0108 18:54:52.960975 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling LogGCOp(50603d14716f4c4bb514a91e103fbf64): free 59148008 bytes of WAL I0108 18:54:52.961055 25056 log_reader.cc:379] T 50603d14716f4c4bb514a91e103fbf64: removed 7 log segments from log reader I0108 18:54:52.961097 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000060 (ops 349773-355722) I0108 18:54:52.961192 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000061 (ops 355723-361788) I0108 18:54:52.961318 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000062 (ops 361789-367735) I0108 18:54:52.961385 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000063 (ops 367736-373641) I0108 18:54:52.961444 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000064 (ops 373642-379602) I0108 18:54:52.961498 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000065 (ops 379603-385496) I0108 18:54:52.961553 25056 log.cc:1062] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-2/wal/wals/50603d14716f4c4bb514a91e103fbf64/wal-000000066 (ops 385497-391485) I0108 18:54:52.979419 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: LogGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.018s user 0.000s sys 0.019s Metrics: {} I0108 18:54:52.979575 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.656984 I0108 18:54:53.205122 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 26.992s user 7.817s sys 0.497s Metrics: {"bytes_written":130183205,"cfile_cache_hit":11354,"cfile_cache_hit_bytes":321991713,"cfile_cache_miss":5930,"cfile_cache_miss_bytes":212798115,"cfile_init":16,"delta_iterators_relevant":12,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":474,"dirs.run_wall_time_us":255771,"drs_written":4,"lbm_read_time_us":6362352,"lbm_reads_1-10_ms":73,"lbm_reads_10-100_ms":141,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":5774,"lbm_write_time_us":10204152,"lbm_writes_1-10_ms":99,"lbm_writes_10-100_ms":230,"lbm_writes_gt_100_ms":12,"lbm_writes_lt_1ms":12643,"num_input_rowsets":5,"rows_written":7330968,"spinlock_wait_cycles":35584,"thread_start_us":183,"threads_started":5,"wal-append.queue_time_us":54} I0108 18:54:53.205279 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.421582 I0108 18:54:53.807682 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 29.142s user 7.761s sys 0.595s Metrics: {"bytes_written":130196950,"cfile_cache_hit":108,"cfile_cache_hit_bytes":10536314,"cfile_cache_miss":17194,"cfile_cache_miss_bytes":524327540,"cfile_init":23,"delta_iterators_relevant":12,"dirs.queue_time_us":16540,"dirs.run_cpu_time_us":486,"dirs.run_wall_time_us":169378,"drs_written":4,"lbm_read_time_us":18458337,"lbm_reads_1-10_ms":205,"lbm_reads_10-100_ms":423,"lbm_reads_gt_100_ms":21,"lbm_reads_lt_1ms":16637,"lbm_write_time_us":323983,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":12981,"num_input_rowsets":6,"rows_written":7330993,"thread_start_us":559,"threads_started":13,"wal-append.queue_time_us":64} I0108 18:54:53.807919 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.544795 I0108 18:55:31.662935 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 37.855s user 7.691s sys 0.645s Metrics: {"bytes_written":130233412,"cfile_cache_hit":94,"cfile_cache_hit_bytes":9938006,"cfile_cache_miss":17176,"cfile_cache_miss_bytes":527742334,"cfile_init":17,"delta_iterators_relevant":8,"dirs.queue_time_us":288,"dirs.run_cpu_time_us":490,"dirs.run_wall_time_us":355554,"drs_written":4,"lbm_read_time_us":21461921,"lbm_reads_1-10_ms":225,"lbm_reads_10-100_ms":478,"lbm_reads_gt_100_ms":26,"lbm_reads_lt_1ms":16515,"lbm_write_time_us":6306127,"lbm_writes_1-10_ms":77,"lbm_writes_10-100_ms":160,"lbm_writes_gt_100_ms":4,"lbm_writes_lt_1ms":12740,"num_input_rowsets":4,"rows_written":7330983,"spinlock_wait_cycles":2176,"thread_start_us":656,"threads_started":13,"wal-append.queue_time_us":51} I0108 18:55:31.663074 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 282486831 bytes on disk I0108 18:55:31.663569 24709 linked_list-test-util.h:876] Done collecting results (7332500 rows in 69891.3ms) I0108 18:55:31.879499 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.216s user 0.001s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":216066,"lbm_reads_1-10_ms":6,"lbm_reads_10-100_ms":4,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":17} I0108 18:55:31.899124 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 38.694s user 7.452s sys 0.527s Metrics: {"bytes_written":130209329,"cfile_cache_hit":5511,"cfile_cache_hit_bytes":156434972,"cfile_cache_miss":11753,"cfile_cache_miss_bytes":381963536,"cfile_init":14,"delta_iterators_relevant":8,"dirs.queue_time_us":307,"dirs.run_cpu_time_us":494,"dirs.run_wall_time_us":353794,"drs_written":4,"lbm_read_time_us":21679162,"lbm_reads_1-10_ms":251,"lbm_reads_10-100_ms":428,"lbm_reads_gt_100_ms":31,"lbm_reads_lt_1ms":11099,"lbm_write_time_us":6670544,"lbm_writes_1-10_ms":76,"lbm_writes_10-100_ms":137,"lbm_writes_gt_100_ms":9,"lbm_writes_lt_1ms":12762,"num_input_rowsets":4,"rows_written":7330795,"spinlock_wait_cycles":3968,"thread_start_us":530,"threads_started":13,"wal-append.queue_time_us":73} I0108 18:55:31.899365 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 240843387 bytes on disk I0108 18:55:32.041118 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.142s user 0.001s sys 0.000s Metrics: {"cfile_init":6,"lbm_read_time_us":141360,"lbm_reads_1-10_ms":7,"lbm_reads_10-100_ms":6,"lbm_reads_lt_1ms":11} I0108 18:55:32.041352 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.220818 I0108 18:55:32.732933 24709 linked_list-test-util.h:667] Snapshot: P: 1578537861018563 usec, L: 1 verified. Result: OK I0108 18:55:32.735658 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578538161018821 usec, L: 1 Expected Rows: 13799750 I0108 18:55:32.869987 24709 linked_list-test-util.h:639] Calling callback on tserver 53135810f2d84759ac110009de750c3d I0108 18:55:34.786262 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 41.807s user 9.432s sys 0.736s Metrics: {"bytes_written":159099738,"cfile_cache_hit":70,"cfile_cache_hit_bytes":4651238,"cfile_cache_miss":21054,"cfile_cache_miss_bytes":650557596,"cfile_init":24,"delta_iterators_relevant":12,"dirs.queue_time_us":426,"dirs.run_cpu_time_us":610,"dirs.run_wall_time_us":276520,"drs_written":5,"lbm_read_time_us":28293156,"lbm_reads_1-10_ms":262,"lbm_reads_10-100_ms":555,"lbm_reads_gt_100_ms":44,"lbm_reads_lt_1ms":20289,"lbm_write_time_us":398322,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":15858,"num_input_rowsets":6,"rows_written":8955861,"thread_start_us":602,"threads_started":14,"wal-append.queue_time_us":77} I0108 18:55:34.786523 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 193270442 bytes on disk I0108 18:55:34.985673 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.199s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":198823,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":6,"lbm_reads_lt_1ms":10} I0108 18:55:34.985767 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.403598 I0108 18:55:40.480232 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 8.439s user 3.707s sys 0.281s Metrics: {"bytes_written":62698960,"cfile_cache_hit":4366,"cfile_cache_hit_bytes":124397503,"cfile_cache_miss":3948,"cfile_cache_miss_bytes":135204915,"cfile_init":6,"delta_iterators_relevant":4,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":48954,"drs_written":2,"lbm_read_time_us":2111663,"lbm_reads_1-10_ms":43,"lbm_reads_10-100_ms":67,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3861,"lbm_write_time_us":1678794,"lbm_writes_1-10_ms":40,"lbm_writes_10-100_ms":59,"lbm_writes_lt_1ms":6153,"num_input_rowsets":2,"rows_written":3529662,"spinlock_wait_cycles":2176,"thread_start_us":113,"threads_started":3,"wal-append.queue_time_us":61} I0108 18:55:40.480401 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 76504204 bytes on disk I0108 18:55:40.480527 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} I0108 18:55:44.639945 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.033369 I0108 18:55:44.802389 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.162s user 0.150s sys 0.010s Metrics: {"bytes_written":1899825,"cfile_init":1,"dirs.queue_time_us":10962,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5181,"lbm_writes_lt_1ms":207,"rows_written":106265,"thread_start_us":67,"threads_started":2,"wal-append.queue_time_us":55} I0108 18:55:44.802568 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling LogGCOp(fdf8e522824640c3ab9385967c93743a): free 8392049 bytes of WAL I0108 18:55:44.802645 24807 log_reader.cc:379] T fdf8e522824640c3ab9385967c93743a: removed 1 log segments from log reader I0108 18:55:44.802688 24807 log.cc:1062] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal/wals/fdf8e522824640c3ab9385967c93743a/wal-000000030 (ops 360723-373162) I0108 18:55:44.805578 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: LogGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.003s user 0.000s sys 0.003s Metrics: {} I0108 18:55:44.805702 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.004925 I0108 18:55:44.984139 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033361 I0108 18:55:45.170064 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.186s user 0.111s sys 0.008s Metrics: {"bytes_written":1900451,"cfile_init":1,"dirs.queue_time_us":131,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":25668,"lbm_writes_1-10_ms":4,"lbm_writes_lt_1ms":203,"rows_written":106226,"thread_start_us":100,"threads_started":2,"wal-append.queue_time_us":71} I0108 18:55:45.170229 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling LogGCOp(dbe8c3d61048482fb282021487b845a6): free 8390036 bytes of WAL I0108 18:55:45.170294 24932 log_reader.cc:379] T dbe8c3d61048482fb282021487b845a6: removed 1 log segments from log reader I0108 18:55:45.170342 24932 log.cc:1062] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6/wal-000000030 (ops 359697-372096) I0108 18:55:45.172317 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: LogGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.002s user 0.000s sys 0.002s Metrics: {} I0108 18:55:45.172399 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.004867 I0108 18:55:50.158576 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 5.353s user 3.104s sys 0.163s Metrics: {"bytes_written":57885594,"cfile_cache_hit":7237,"cfile_cache_hit_bytes":206599855,"cfile_cache_miss":451,"cfile_cache_miss_bytes":33203273,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":426,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":217898,"drs_written":2,"lbm_read_time_us":641223,"lbm_reads_1-10_ms":9,"lbm_reads_10-100_ms":17,"lbm_reads_lt_1ms":457,"lbm_write_time_us":145836,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":5773,"num_input_rowsets":3,"rows_written":3258134,"thread_start_us":698,"threads_started":11,"wal-append.queue_time_us":75} I0108 18:55:50.158777 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 70494002 bytes on disk I0108 18:55:50.158913 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":14,"lbm_reads_lt_1ms":8} I0108 18:55:52.281006 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 17.295s user 6.133s sys 0.480s Metrics: {"bytes_written":96464063,"cfile_cache_hit":68,"cfile_cache_hit_bytes":6166505,"cfile_cache_miss":12730,"cfile_cache_miss_bytes":392588861,"cfile_init":15,"delta_iterators_relevant":8,"dirs.queue_time_us":883,"dirs.run_cpu_time_us":330,"dirs.run_wall_time_us":127752,"drs_written":3,"lbm_read_time_us":9288182,"lbm_reads_1-10_ms":157,"lbm_reads_10-100_ms":258,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":12367,"lbm_write_time_us":238961,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":9617,"num_input_rowsets":4,"rows_written":5430368,"thread_start_us":628,"threads_started":12,"wal-append.queue_time_us":56} I0108 18:55:52.281213 25162 maintenance_manager.cc:326] P 61b86269d42a403f94c0a049429fa247: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 117703380 bytes on disk I0108 18:55:52.620559 25056 maintenance_manager.cc:527] P 61b86269d42a403f94c0a049429fa247: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.339s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":339088,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":8} I0108 18:55:55.369349 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 10.197s user 3.518s sys 0.312s Metrics: {"bytes_written":57884267,"cfile_cache_hit":19,"cfile_cache_hit_bytes":79484,"cfile_cache_miss":7669,"cfile_cache_miss_bytes":239680422,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":370,"dirs.run_wall_time_us":78713,"drs_written":2,"lbm_read_time_us":4199697,"lbm_reads_1-10_ms":79,"lbm_reads_10-100_ms":117,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":7517,"lbm_write_time_us":1367895,"lbm_writes_1-10_ms":26,"lbm_writes_10-100_ms":27,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":5720,"num_input_rowsets":3,"rows_written":3257656,"spinlock_wait_cycles":768,"thread_start_us":533,"threads_started":11,"wal-append.queue_time_us":70} I0108 18:55:55.369550 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 70471258 bytes on disk I0108 18:55:55.369695 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 18:56:03.625953 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(fdf8e522824640c3ab9385967c93743a): perf score=0.033363 I0108 18:56:03.703532 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.077s user 0.068s sys 0.005s Metrics: {"bytes_written":1087733,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":83,"dirs.run_wall_time_us":14409,"drs_written":1,"lbm_read_time_us":4,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2868,"lbm_writes_lt_1ms":125,"rows_written":60466,"thread_start_us":71,"threads_started":2,"wal-append.queue_time_us":52} I0108 18:56:03.703691 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 1159490 bytes on disk I0108 18:56:03.703814 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0108 18:56:03.703866 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a): perf score=1.005387 I0108 18:56:06.921921 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling FlushMRSOp(dbe8c3d61048482fb282021487b845a6): perf score=0.033337 I0108 18:56:07.008805 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: FlushMRSOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.087s user 0.051s sys 0.001s Metrics: {"bytes_written":996247,"cfile_init":1,"dirs.queue_time_us":83,"drs_written":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1860,"lbm_writes_lt_1ms":117,"rows_written":55305,"thread_start_us":73,"threads_started":2,"wal-append.queue_time_us":56} I0108 18:56:07.008975 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6): perf score=1.005371 I0108 18:56:12.539384 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: CompactRowSetsOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 5.530s user 3.219s sys 0.193s Metrics: {"bytes_written":57884343,"cfile_cache_hit":7351,"cfile_cache_hit_bytes":210016753,"cfile_cache_miss":333,"cfile_cache_miss_bytes":30029479,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":523,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":130339,"drs_written":2,"lbm_read_time_us":287913,"lbm_reads_1-10_ms":7,"lbm_reads_10-100_ms":7,"lbm_reads_lt_1ms":351,"lbm_write_time_us":1163801,"lbm_writes_1-10_ms":10,"lbm_writes_10-100_ms":24,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":5738,"num_input_rowsets":3,"rows_written":3257656,"spinlock_wait_cycles":384,"thread_start_us":642,"threads_started":11,"wal-append.queue_time_us":79} I0108 18:56:12.542366 24913 maintenance_manager.cc:326] P c6ad452ff3b64d949298a50527e1fbb0: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 70472659 bytes on disk I0108 18:56:12.601291 24807 maintenance_manager.cc:527] P c6ad452ff3b64d949298a50527e1fbb0: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.059s user 0.000s sys 0.001s Metrics: {"cfile_init":2,"lbm_read_time_us":58792,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":7} I0108 18:56:16.364003 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 12.660s user 3.583s sys 0.324s Metrics: {"bytes_written":57887092,"cfile_cache_hit":19,"cfile_cache_hit_bytes":79657,"cfile_cache_miss":7665,"cfile_cache_miss_bytes":239983421,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":540,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":110795,"drs_written":2,"lbm_read_time_us":7522966,"lbm_reads_1-10_ms":133,"lbm_reads_10-100_ms":259,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":7313,"lbm_write_time_us":659804,"lbm_writes_1-10_ms":16,"lbm_writes_10-100_ms":18,"lbm_writes_lt_1ms":5741,"num_input_rowsets":3,"rows_written":3258134,"spinlock_wait_cycles":3328,"thread_start_us":502,"threads_started":11,"wal-append.queue_time_us":64} I0108 18:56:16.364185 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 70493419 bytes on disk I0108 18:56:16.425895 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.062s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":61505,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 18:56:24.167577 24709 linked_list-test-util.h:876] Done collecting results (13799750 rows in 51297.6ms) I0108 18:56:25.873203 24709 linked_list-test-util.h:667] Snapshot: P: 1578538161018821 usec, L: 1 verified. Result: OK I0108 18:56:25.873750 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578538461013192 usec, L: 1 Expected Rows: 19548500 I0108 18:56:25.920274 24709 linked_list-test-util.h:639] Calling callback on tserver 53135810f2d84759ac110009de750c3d I0108 18:56:26.183421 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling FlushMRSOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.033350 I0108 18:56:26.185621 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: FlushMRSOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"bytes_written":6384,"cfile_init":1,"dirs.queue_time_us":73,"dirs.run_cpu_time_us":81,"dirs.run_wall_time_us":1004,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":102,"lbm_writes_lt_1ms":26,"rows_written":102,"thread_start_us":55,"threads_started":2,"wal-append.queue_time_us":37} I0108 18:56:26.185714 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.589391 I0108 18:56:37.237455 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 11.052s user 6.721s sys 0.531s Metrics: {"bytes_written":135074716,"cfile_cache_hit":4758,"cfile_cache_hit_bytes":165543390,"cfile_cache_miss":13174,"cfile_cache_miss_bytes":397509916,"cfile_init":13,"delta_iterators_relevant":10,"dirs.queue_time_us":4520,"dirs.run_cpu_time_us":566,"dirs.run_wall_time_us":177357,"drs_written":5,"lbm_read_time_us":1829839,"lbm_reads_1-10_ms":138,"lbm_reads_10-100_ms":57,"lbm_reads_lt_1ms":13031,"lbm_write_time_us":466587,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":13481,"num_input_rowsets":5,"rows_written":7602427,"spinlock_wait_cycles":384,"thread_start_us":167,"threads_started":5,"wal-append.queue_time_us":74} I0108 18:56:37.237574 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 164786479 bytes on disk I0108 18:56:37.237730 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":27,"lbm_reads_lt_1ms":20} I0108 18:56:37.237800 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.270483 I0108 18:56:42.290122 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 5.052s user 3.057s sys 0.249s Metrics: {"bytes_written":62698876,"cfile_cache_hit":19,"cfile_cache_hit_bytes":86381,"cfile_cache_miss":8315,"cfile_cache_miss_bytes":261306029,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":81901,"drs_written":2,"lbm_read_time_us":1064608,"lbm_reads_1-10_ms":68,"lbm_reads_10-100_ms":38,"lbm_reads_lt_1ms":8253,"lbm_write_time_us":126638,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":6251,"num_input_rowsets":3,"rows_written":3529745,"thread_start_us":104,"threads_started":3,"wal-append.queue_time_us":70} I0108 18:56:42.290278 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 76505085 bytes on disk I0108 18:56:42.290410 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":12,"lbm_reads_lt_1ms":8} I0108 18:56:42.290496 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64): perf score=0.214116 I0108 18:56:48.687213 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: CompactRowSetsOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 6.397s user 3.667s sys 0.270s Metrics: {"bytes_written":67547191,"cfile_cache_hit":255,"cfile_cache_hit_bytes":42549297,"cfile_cache_miss":8701,"cfile_cache_miss_bytes":238983703,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":456,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":156360,"drs_written":3,"lbm_read_time_us":1374454,"lbm_reads_1-10_ms":52,"lbm_reads_10-100_ms":46,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":8630,"lbm_write_time_us":603701,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":8,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":6741,"num_input_rowsets":2,"rows_written":3801251,"thread_start_us":584,"threads_started":11,"wal-append.queue_time_us":66} I0108 18:56:48.687444 25038 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 82390849 bytes on disk I0108 18:56:48.721354 24932 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.034s user 0.001s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":33714,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":9} I0108 18:57:04.538600 24709 linked_list-test-util.h:876] Done collecting results (19548500 rows in 38618.3ms) I0108 18:57:06.988999 24709 linked_list-test-util.h:667] Snapshot: P: 1578538461013192 usec, L: 1 verified. Result: OK I0108 18:57:06.989514 24709 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 19548500 I0108 18:57:41.455158 24709 linked_list-test-util.h:876] Done collecting results (19548500 rows in 34223.3ms) I0108 18:57:43.905270 24709 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0108 18:57:43.905519 24709 linked_list-test-util.h:801] Successfully verified 19548500 rows I0108 18:57:43.910461 24709 linked_list-test.cc:194] Successfully verified 19548500 rows before killing any servers. I0108 18:57:43.911960 24709 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0108 18:57:43.912212 24709 ts_itest-base.cc:233] Waiting for 3 tablets on tserver c6ad452ff3b64d949298a50527e1fbb0 to finish bootstrapping I0108 18:57:43.913620 24709 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 53135810f2d84759ac110009de750c3d to finish bootstrapping I0108 18:57:43.915143 24709 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 61b86269d42a403f94c0a049429fa247 to finish bootstrapping I0108 18:57:43.916654 24709 linked_list-test.cc:199] Will restart the tablet server during verification scan. I0108 18:57:43.917011 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578537861018563 usec, L: 1 Expected Rows: 7332500 I0108 18:57:43.983896 24709 linked_list-test-util.h:639] Calling callback on tserver 53135810f2d84759ac110009de750c3d I0108 18:57:43.983966 24709 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24916 I0108 18:57:44.036831 24709 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.24.33.66:33490 --local_ip_for_outbound_sockets=127.24.33.66 --webserver_port=50336 --webserver_interface=127.24.33.66 --tserver_master_addrs=127.24.33.126:55767 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.24.33.84:49727 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 18:57:44.048756 30468 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 18:57:44.048866 30468 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 18:57:44.049676 30468 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 18:57:44.049707 30468 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.33.66 I0108 18:57:44.050763 30468 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.24.33.84:49727 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.24.33.66:33490 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/info.pb --webserver_interface=127.24.33.66 --webserver_port=50336 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.24.33.126:55767 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.30468 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.24.33.66 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 4dca6b02a5e7b2e589646e79bc6e509f27c13c3d build type RELEASE built by jenkins at 08 Jan 2020 18:26:25 PST on quasar-luegrb-1.vpc.cloudera.com I0108 18:57:44.052893 30468 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 18:57:44.054069 30468 hybrid_clock.cc:186] HybridClock initialized: now 1578538664054058 us; error 16 us; skew 500 ppm I0108 18:57:44.054153 30468 fs_manager.cc:263] Metadata directory not provided I0108 18:57:44.054180 30468 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 18:57:44.056394 30468 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s I0108 18:57:44.056546 30468 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 18:57:44.056610 30468 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 18:57:44.058737 30468 fs_manager.cc:434] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0108 18:57:44.058918 30468 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal uuid: "53135810f2d84759ac110009de750c3d" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 18:57:44.058955 30468 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/data Total live blocks: 72 Total live bytes: 770762741 Total live bytes (after alignment): 770945024 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 18:57:44.096596 30468 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 18:57:44.096834 30468 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0108 18:57:44.097658 30468 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0108 18:57:44.097676 30468 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0108 18:57:44.097978 30490 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Bootstrap starting. I0108 18:57:44.098269 30468 ts_tablet_manager.cc:415] Registered 3 tablets I0108 18:57:44.102869 30468 rpc_server.cc:211] RPC server started. Bound to: 127.24.33.66:33490 I0108 18:57:44.103204 30468 webserver.cc:382] Webserver started at http://127.24.33.66:50336/ using document root and password file I0108 18:57:44.103432 30468 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/data/info.pb I0108 18:57:44.106088 30595 heartbeater.cc:322] Connected to a master server at 127.24.33.126:55767 I0108 18:57:44.106106 30595 heartbeater.cc:406] Registering TS with master... I0108 18:57:44.106138 30595 heartbeater.cc:455] Master 127.24.33.126:55767 requested a full tablet report, sending... I0108 18:57:44.106335 24743 ts_manager.cc:192] Re-registered known tserver with Master: 53135810f2d84759ac110009de750c3d (127.24.33.66:33490) I0108 18:57:44.106803 24743 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.24.33.66:41457 I0108 18:57:44.110368 24709 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 30468 I0108 18:57:44.114456 30512 tablet_service.cc:2571] Scan: Not found: Scanner b3a350c9014a45b4a36b599d1823f843 not found (it may have expired): call sequence id=1, remote={username='jenkins'} at 127.24.33.66:58641 W0108 18:57:44.114626 24709 client.cc:1692] Attempting to retry Scanner { table: TestTable, tablet: 50603d14716f4c4bb514a91e103fbf64, projection: ( rand_key INT64 NOT NULL, link_to INT64 NOT NULL, updated BOOL NOT NULL, PRIMARY KEY () ), scan_spec: } elsewhere. I0108 18:57:44.203580 30490 log.cc:815] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Log is configured to *not* fsync() on all Append() calls W0108 18:57:44.204071 25050 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer 53135810f2d84759ac110009de750c3d (127.24.33.66:33490): Couldn't send request to peer 53135810f2d84759ac110009de750c3d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry. I0108 18:57:44.343166 30490 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1981 overwritten=0 applied=1981 ignored=1981} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates I0108 18:57:44.343274 30490 log.cc:1202] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/50603d14716f4c4bb514a91e103fbf64.recovery-1578538664343213 I0108 18:57:44.343286 30490 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Bootstrap complete. I0108 18:57:44.343328 30490 ts_tablet_manager.cc:1105] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Time spent bootstrapping tablet: real 0.245s user 0.147s sys 0.028s I0108 18:57:44.343986 30490 raft_consensus.cc:341] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:57:44.344009 30490 raft_consensus.cc:712] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53135810f2d84759ac110009de750c3d, State: Initialized, Role: FOLLOWER I0108 18:57:44.344077 30490 consensus_queue.cc:229] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 391854, Last appended: 1.391854, Last appended by leader: 391854, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:57:44.344246 30595 heartbeater.cc:447] Master 127.24.33.126:55767 was elected leader, sending a full tablet report... I0108 18:57:44.344260 30490 ts_tablet_manager.cc:1133] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0108 18:57:44.344285 30490 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Bootstrap starting. I0108 18:57:44.354169 30596 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64): 282484273 bytes on disk I0108 18:57:44.502677 30489 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(50603d14716f4c4bb514a91e103fbf64) complete. Timing: real 0.148s user 0.000s sys 0.001s Metrics: {"cfile_init":8,"lbm_read_time_us":148085,"lbm_reads_1-10_ms":5,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":22} I0108 18:57:44.860828 30490 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11312 overwritten=0 applied=11312 ignored=11312} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0108 18:57:44.860900 30490 log.cc:1202] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/dbe8c3d61048482fb282021487b845a6.recovery-1578538664860865 I0108 18:57:44.860906 30490 tablet_bootstrap.cc:439] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Bootstrap complete. I0108 18:57:44.860930 30490 ts_tablet_manager.cc:1105] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Time spent bootstrapping tablet: real 0.517s user 0.338s sys 0.097s I0108 18:57:44.861016 30490 raft_consensus.cc:341] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:44.861030 30490 raft_consensus.cc:712] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53135810f2d84759ac110009de750c3d, State: Initialized, Role: FOLLOWER I0108 18:57:44.861075 30490 consensus_queue.cc:229] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 383408, Last appended: 1.383408, Last appended by leader: 383408, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:44.861163 30490 ts_tablet_manager.cc:1133] T dbe8c3d61048482fb282021487b845a6 P 53135810f2d84759ac110009de750c3d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0108 18:57:44.861191 30490 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Bootstrap starting. I0108 18:57:45.003226 30596 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6): 70471258 bytes on disk I0108 18:57:45.040925 30489 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(dbe8c3d61048482fb282021487b845a6) complete. Timing: real 0.038s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":37447,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 18:57:45.105332 30606 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53135810f2d84759ac110009de750c3d) I0108 18:57:45.105412 30606 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:45.105540 30606 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 18:57:45.105684 30552 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 2 candidate_status { last_received { term: 1 index: 383604 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true I0108 18:57:45.105703 24867 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 2 candidate_status { last_received { term: 1 index: 383604 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" is_pre_election: true W0108 18:57:45.105814 25050 leader_election.cc:346] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53135810f2d84759ac110009de750c3d (127.24.33.66:33490): Illegal state: must be running to vote when last-logged opid is not known I0108 18:57:45.105908 25049 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b86269d42a403f94c0a049429fa247; no voters: 53135810f2d84759ac110009de750c3d, c6ad452ff3b64d949298a50527e1fbb0 I0108 18:57:45.106015 30606 raft_consensus.cc:2644] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0108 18:57:45.163856 30609 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53135810f2d84759ac110009de750c3d) I0108 18:57:45.164038 30609 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:45.164256 30609 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:57:45.167645 25116 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 383604 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" is_pre_election: true I0108 18:57:45.167678 25116 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ad452ff3b64d949298a50527e1fbb0 in term 1. I0108 18:57:45.167778 24798 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61b86269d42a403f94c0a049429fa247, c6ad452ff3b64d949298a50527e1fbb0; no voters: I0108 18:57:45.167824 30609 raft_consensus.cc:2696] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0108 18:57:45.167837 30609 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 53135810f2d84759ac110009de750c3d) I0108 18:57:45.167845 30609 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Advancing to term 2 I0108 18:57:45.167909 30552 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 383604 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true W0108 18:57:45.168009 24800 leader_election.cc:346] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 53135810f2d84759ac110009de750c3d (127.24.33.66:33490): Illegal state: must be running to vote when last-logged opid is not known I0108 18:57:45.168501 30609 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:45.168551 30609 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 election: Requested vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:57:45.168663 25116 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 383604 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" I0108 18:57:45.168684 25116 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Advancing to term 2 I0108 18:57:45.168694 30552 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 383604 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" W0108 18:57:45.168817 24800 leader_election.cc:346] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 53135810f2d84759ac110009de750c3d (127.24.33.66:33490): Illegal state: must be running to vote when last-logged opid is not known I0108 18:57:45.170418 25116 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ad452ff3b64d949298a50527e1fbb0 in term 2. I0108 18:57:45.170503 24798 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 61b86269d42a403f94c0a049429fa247, c6ad452ff3b64d949298a50527e1fbb0; no voters: 53135810f2d84759ac110009de750c3d I0108 18:57:45.170553 30609 raft_consensus.cc:2696] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 2 FOLLOWER]: Leader election won for term 2 I0108 18:57:45.170644 30609 raft_consensus.cc:672] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [term 2 LEADER]: Becoming Leader. State: Replica: c6ad452ff3b64d949298a50527e1fbb0, State: Running, Role: LEADER I0108 18:57:45.170714 30609 consensus_queue.cc:206] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383604, Committed index: 383604, Last appended: 1.383604, Last appended by leader: 383604, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:45.171171 24743 catalog_manager.cc:4143] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 reported cstate change: term changed from 1 to 2, leader changed from 53135810f2d84759ac110009de750c3d (127.24.33.66) to c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65). New cstate: current_term: 2 leader_uuid: "c6ad452ff3b64d949298a50527e1fbb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } health_report { overall_health: UNKNOWN } } } I0108 18:57:45.264235 30490 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11653 overwritten=0 applied=11653 ignored=11653} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0108 18:57:45.264333 30490 log.cc:1202] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-1/wal/wals/fdf8e522824640c3ab9385967c93743a.recovery-1578538665264298 I0108 18:57:45.264346 30490 tablet_bootstrap.cc:439] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Bootstrap complete. I0108 18:57:45.264367 30490 ts_tablet_manager.cc:1105] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Time spent bootstrapping tablet: real 0.403s user 0.298s sys 0.075s I0108 18:57:45.264444 30490 raft_consensus.cc:341] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:45.264457 30490 raft_consensus.cc:712] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53135810f2d84759ac110009de750c3d, State: Initialized, Role: FOLLOWER I0108 18:57:45.264488 30490 consensus_queue.cc:229] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 383604, Last appended: 1.383604, Last appended by leader: 383604, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 18:57:45.264533 30490 ts_tablet_manager.cc:1133] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 18:57:45.291272 30596 maintenance_manager.cc:326] P 53135810f2d84759ac110009de750c3d: Scheduling UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a): 70493419 bytes on disk I0108 18:57:45.294557 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.306877 I0108 18:57:45.295311 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":441,"delete_count":0,"lbm_write_time_us":29,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1} I0108 18:57:45.320799 30489 maintenance_manager.cc:527] P 53135810f2d84759ac110009de750c3d: UndoDeltaBlockGCOp(fdf8e522824640c3ab9385967c93743a) complete. Timing: real 0.029s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":29274,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":5} I0108 18:57:45.362346 30606 raft_consensus.cc:477] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53135810f2d84759ac110009de750c3d) I0108 18:57:45.362424 30606 raft_consensus.cc:499] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:57:45.362489 30606 leader_election.cc:293] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 18:57:45.362630 30552 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 2 candidate_status { last_received { term: 1 index: 391854 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true I0108 18:57:45.362664 30552 raft_consensus.cc:2396] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61b86269d42a403f94c0a049429fa247 in term 1. I0108 18:57:45.362686 24867 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 2 candidate_status { last_received { term: 1 index: 391854 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" is_pre_election: true I0108 18:57:45.362814 25050 leader_election.cc:307] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: I0108 18:57:45.362877 30606 raft_consensus.cc:2696] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0108 18:57:45.362890 30606 raft_consensus.cc:477] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 53135810f2d84759ac110009de750c3d) I0108 18:57:45.362898 30606 raft_consensus.cc:2949] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 1 FOLLOWER]: Advancing to term 2 I0108 18:57:45.363948 30606 raft_consensus.cc:499] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:57:45.363999 30606 leader_election.cc:293] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 election: Requested vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 18:57:45.364095 24867 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 2 candidate_status { last_received { term: 1 index: 391854 } } ignore_live_leader: false dest_uuid: "c6ad452ff3b64d949298a50527e1fbb0" I0108 18:57:45.364095 30552 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 2 candidate_status { last_received { term: 1 index: 391854 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" I0108 18:57:45.364111 30552 raft_consensus.cc:2949] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Advancing to term 2 I0108 18:57:45.365190 30552 raft_consensus.cc:2396] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61b86269d42a403f94c0a049429fa247 in term 2. I0108 18:57:45.365268 25050 leader_election.cc:307] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: c6ad452ff3b64d949298a50527e1fbb0 I0108 18:57:45.365337 30606 raft_consensus.cc:2696] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Leader election won for term 2 I0108 18:57:45.365388 30606 raft_consensus.cc:672] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [term 2 LEADER]: Becoming Leader. State: Replica: 61b86269d42a403f94c0a049429fa247, State: Running, Role: LEADER I0108 18:57:45.365442 30606 consensus_queue.cc:206] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 391854, Committed index: 391854, Last appended: 1.391854, Last appended by leader: 391854, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:57:45.365753 24743 catalog_manager.cc:4143] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 reported cstate change: term changed from 1 to 2, leader changed from 53135810f2d84759ac110009de750c3d (127.24.33.66) to 61b86269d42a403f94c0a049429fa247 (127.24.33.67). New cstate: current_term: 2 leader_uuid: "61b86269d42a403f94c0a049429fa247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } health_report { overall_health: UNKNOWN } } } I0108 18:57:45.391695 30609 raft_consensus.cc:477] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53135810f2d84759ac110009de750c3d) I0108 18:57:45.391757 30609 raft_consensus.cc:499] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } I0108 18:57:45.391804 30609 leader_election.cc:293] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 18:57:45.391975 25116 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 391854 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" is_pre_election: true I0108 18:57:45.391988 30552 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "50603d14716f4c4bb514a91e103fbf64" candidate_uuid: "c6ad452ff3b64d949298a50527e1fbb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 391854 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true I0108 18:57:45.392009 30552 raft_consensus.cc:2321] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6ad452ff3b64d949298a50527e1fbb0 in current term 2: Already voted for candidate 61b86269d42a403f94c0a049429fa247 in this term. I0108 18:57:45.392132 24800 leader_election.cc:307] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ad452ff3b64d949298a50527e1fbb0; no voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247 I0108 18:57:45.392181 30609 raft_consensus.cc:2949] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 1 FOLLOWER]: Advancing to term 2 I0108 18:57:45.393153 30609 raft_consensus.cc:2644] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0108 18:57:45.547327 25116 raft_consensus.cc:1195] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Refusing update from remote peer c6ad452ff3b64d949298a50527e1fbb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 383604. Preceding OpId from leader: term: 2 index: 383605. (index mismatch) I0108 18:57:45.547577 30609 consensus_queue.cc:982] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383605, Last known committed idx: 383604, Time since last communication: 0.000s I0108 18:57:45.548456 30552 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 1 FOLLOWER]: Advancing to term 2 I0108 18:57:45.549746 30552 raft_consensus.cc:1195] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Refusing update from remote peer c6ad452ff3b64d949298a50527e1fbb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 383604. Preceding OpId from leader: term: 2 index: 383605. (index mismatch) I0108 18:57:45.549886 30624 consensus_queue.cc:982] T fdf8e522824640c3ab9385967c93743a P c6ad452ff3b64d949298a50527e1fbb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383605, Last known committed idx: 383604, Time since last communication: 0.000s I0108 18:57:45.821790 30552 raft_consensus.cc:1195] T 50603d14716f4c4bb514a91e103fbf64 P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Refusing update from remote peer 61b86269d42a403f94c0a049429fa247: Log matching property violated. Preceding OpId in replica: term: 1 index: 391854. Preceding OpId from leader: term: 2 index: 391855. (index mismatch) I0108 18:57:45.822036 30606 consensus_queue.cc:982] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391855, Last known committed idx: 391854, Time since last communication: 0.000s I0108 18:57:45.822770 24867 raft_consensus.cc:1195] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0 [term 2 FOLLOWER]: Refusing update from remote peer 61b86269d42a403f94c0a049429fa247: Log matching property violated. Preceding OpId in replica: term: 1 index: 391854. Preceding OpId from leader: term: 2 index: 391855. (index mismatch) I0108 18:57:45.822897 30630 consensus_queue.cc:982] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 391855, Last known committed idx: 391854, Time since last communication: 0.000s I0108 18:58:06.289485 24709 linked_list-test-util.h:876] Done collecting results (7332500 rows in 22305.6ms) I0108 18:58:07.151170 24709 linked_list-test-util.h:667] Snapshot: P: 1578537861018563 usec, L: 1 verified. Result: OK I0108 18:58:07.151769 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578538161018821 usec, L: 1 Expected Rows: 13799750 I0108 18:58:07.152911 24709 linked_list-test-util.h:639] Calling callback on tserver 61b86269d42a403f94c0a049429fa247 I0108 18:58:44.100304 24709 linked_list-test-util.h:876] Done collecting results (13799750 rows in 36947.4ms) I0108 18:58:45.820745 24709 linked_list-test-util.h:667] Snapshot: P: 1578538161018821 usec, L: 1 verified. Result: OK I0108 18:58:45.821255 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578538461013192 usec, L: 1 Expected Rows: 19548500 I0108 18:58:45.851281 24709 linked_list-test-util.h:639] Calling callback on tserver c6ad452ff3b64d949298a50527e1fbb0 I0108 18:59:23.151772 24709 linked_list-test-util.h:876] Done collecting results (19548500 rows in 37300.5ms) I0108 18:59:25.614094 24709 linked_list-test-util.h:667] Snapshot: P: 1578538461013192 usec, L: 1 verified. Result: OK I0108 18:59:25.614648 24709 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 19548500 I0108 18:59:45.380034 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033357 I0108 18:59:45.381129 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":434,"delete_count":0,"lbm_write_time_us":37,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":29,"threads_started":1,"update_count":1,"wal-append.queue_time_us":51} I0108 18:59:45.381186 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.283630 I0108 18:59:45.382055 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"cfile_cache_hit":5,"cfile_cache_hit_bytes":2957,"cfile_cache_miss":2,"cfile_cache_miss_bytes":333,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":43,"lbm_read_time_us":18,"lbm_reads_lt_1ms":10,"lbm_write_time_us":46,"lbm_writes_lt_1ms":9,"reinsert_count":0,"spinlock_wait_cycles":384,"thread_start_us":40,"threads_started":2,"update_count":2} I0108 18:59:45.382128 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000 I0108 18:59:45.382184 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0108 19:00:00.372344 24709 linked_list-test-util.h:876] Done collecting results (19548500 rows in 34368.3ms) I0108 19:00:02.805306 24709 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0108 19:00:02.805369 24709 linked_list-test-util.h:801] Successfully verified 19548500 rows I0108 19:00:02.805440 24709 linked_list-test.cc:204] Done with tserver restart test. I0108 19:00:02.809904 24709 linked_list-test.cc:210] Will kill the tablet server during verification scan. I0108 19:00:02.810282 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578537861018563 usec, L: 1 Expected Rows: 7332500 I0108 19:00:02.811197 24709 linked_list-test-util.h:639] Calling callback on tserver c6ad452ff3b64d949298a50527e1fbb0 I0108 19:00:02.811256 24709 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24790 I0108 19:00:02.854693 24709 meta_cache.cc:1038] Marking tablet server c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) as failed. W0108 19:00:02.854751 24709 meta_cache.cc:234] Tablet 50603d14716f4c4bb514a91e103fbf64: Replica c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111) W0108 19:00:02.854773 24709 client.cc:1692] Attempting to retry Scanner { table: TestTable, tablet: 50603d14716f4c4bb514a91e103fbf64, projection: ( rand_key INT64 NOT NULL, link_to INT64 NOT NULL, updated BOOL NOT NULL, PRIMARY KEY () ), scan_spec: } elsewhere. W0108 19:00:03.147378 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0108 19:00:03.343072 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0108 19:00:03.849344 30804 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6ad452ff3b64d949298a50527e1fbb0) I0108 19:00:03.849413 30804 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 19:00:03.849515 30804 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 53135810f2d84759ac110009de750c3d (127.24.33.66:33490), c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) I0108 19:00:03.849655 30551 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "61b86269d42a403f94c0a049429fa247" candidate_term: 3 candidate_status { last_received { term: 2 index: 383605 } } ignore_live_leader: false dest_uuid: "53135810f2d84759ac110009de750c3d" is_pre_election: true W0108 19:00:03.849697 25049 leader_election.cc:339] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111) I0108 19:00:03.849797 25050 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b86269d42a403f94c0a049429fa247; no voters: 53135810f2d84759ac110009de750c3d, c6ad452ff3b64d949298a50527e1fbb0 I0108 19:00:03.849859 30804 raft_consensus.cc:2644] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0108 19:00:04.215574 30805 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6ad452ff3b64d949298a50527e1fbb0) I0108 19:00:04.215656 30805 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 19:00:04.215778 30805 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) W0108 19:00:04.216012 30482 leader_election.cc:339] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111) I0108 19:00:04.217977 25116 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 383605 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" is_pre_election: true I0108 19:00:04.218003 25116 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 2. I0108 19:00:04.218082 30481 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: c6ad452ff3b64d949298a50527e1fbb0 I0108 19:00:04.218135 30805 raft_consensus.cc:2696] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Leader pre-election won for term 3 I0108 19:00:04.218150 30805 raft_consensus.cc:477] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Starting leader election (detected failure of leader c6ad452ff3b64d949298a50527e1fbb0) I0108 19:00:04.218158 30805 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 2 FOLLOWER]: Advancing to term 3 I0108 19:00:04.218721 30805 raft_consensus.cc:499] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 19:00:04.218765 30805 leader_election.cc:293] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 3 election: Requested vote from peers c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934), 61b86269d42a403f94c0a049429fa247 (127.24.33.67:35103) I0108 19:00:04.218863 25116 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "fdf8e522824640c3ab9385967c93743a" candidate_uuid: "53135810f2d84759ac110009de750c3d" candidate_term: 3 candidate_status { last_received { term: 2 index: 383605 } } ignore_live_leader: false dest_uuid: "61b86269d42a403f94c0a049429fa247" I0108 19:00:04.218878 25116 raft_consensus.cc:2949] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 2 FOLLOWER]: Advancing to term 3 W0108 19:00:04.218889 30482 leader_election.cc:339] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111) I0108 19:00:04.221024 25116 raft_consensus.cc:2396] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53135810f2d84759ac110009de750c3d in term 3. I0108 19:00:04.221101 30481 leader_election.cc:307] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53135810f2d84759ac110009de750c3d, 61b86269d42a403f94c0a049429fa247; no voters: c6ad452ff3b64d949298a50527e1fbb0 I0108 19:00:04.221148 30805 raft_consensus.cc:2696] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 3 FOLLOWER]: Leader election won for term 3 I0108 19:00:04.221216 30805 raft_consensus.cc:672] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [term 3 LEADER]: Becoming Leader. State: Replica: 53135810f2d84759ac110009de750c3d, State: Running, Role: LEADER I0108 19:00:04.221274 30805 consensus_queue.cc:206] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383605, Committed index: 383605, Last appended: 2.383605, Last appended by leader: 383605, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } } I0108 19:00:04.221675 24746 catalog_manager.cc:4143] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d reported cstate change: term changed from 2 to 3, leader changed from c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65) to 53135810f2d84759ac110009de750c3d (127.24.33.66). New cstate: current_term: 3 leader_uuid: "53135810f2d84759ac110009de750c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53135810f2d84759ac110009de750c3d" member_type: VOTER last_known_addr { host: "127.24.33.66" port: 33490 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ad452ff3b64d949298a50527e1fbb0" member_type: VOTER last_known_addr { host: "127.24.33.65" port: 55934 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 } health_report { overall_health: UNKNOWN } } } I0108 19:00:04.668687 25116 raft_consensus.cc:1195] T fdf8e522824640c3ab9385967c93743a P 61b86269d42a403f94c0a049429fa247 [term 3 FOLLOWER]: Refusing update from remote peer 53135810f2d84759ac110009de750c3d: Log matching property violated. Preceding OpId in replica: term: 2 index: 383605. Preceding OpId from leader: term: 3 index: 383606. (index mismatch) I0108 19:00:04.668937 30805 consensus_queue.cc:982] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b86269d42a403f94c0a049429fa247" member_type: VOTER last_known_addr { host: "127.24.33.67" port: 35103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383606, Last known committed idx: 383605, Time since last communication: 0.000s W0108 19:00:04.669813 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0108 19:00:05.510795 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0108 19:00:05.956418 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0108 19:00:07.220021 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0108 19:00:07.777657 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0108 19:00:08.307332 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0108 19:00:09.952090 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0108 19:00:09.989089 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0108 19:00:10.827406 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0108 19:00:12.311389 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0108 19:00:12.663727 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0108 19:00:13.291499 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0108 19:00:14.583073 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0108 19:00:15.317459 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0108 19:00:15.663702 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0108 19:00:17.145876 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0108 19:00:17.957842 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0108 19:00:18.279011 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0108 19:00:19.590114 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0108 19:00:20.426676 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0108 19:00:20.750141 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0108 19:00:22.244365 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0108 19:00:22.923156 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0108 19:00:23.181211 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. I0108 19:00:23.279217 24709 linked_list-test-util.h:876] Done collecting results (7332500 rows in 20468ms) I0108 19:00:24.135826 24709 linked_list-test-util.h:667] Snapshot: P: 1578537861018563 usec, L: 1 verified. Result: OK I0108 19:00:24.136379 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578538161018821 usec, L: 1 Expected Rows: 13799750 I0108 19:00:24.137480 24709 linked_list-test-util.h:639] Calling callback on tserver 61b86269d42a403f94c0a049429fa247 W0108 19:00:24.889355 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0108 19:00:25.613363 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0108 19:00:25.706998 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0108 19:00:27.594336 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0108 19:00:28.162535 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0108 19:00:28.164662 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0108 19:00:30.002315 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0108 19:00:30.583053 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0108 19:00:30.593127 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0108 19:00:32.478596 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0108 19:00:32.979609 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0108 19:00:33.059505 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0108 19:00:34.859666 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0108 19:00:35.693526 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0108 19:00:35.776984 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0108 19:00:37.571648 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0108 19:00:37.983628 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0108 19:00:37.995254 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0108 19:00:39.699146 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0108 19:00:40.150935 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0108 19:00:40.631325 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0108 19:00:42.303598 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0108 19:00:42.368311 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0108 19:00:43.013751 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0108 19:00:44.583446 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0108 19:00:44.710968 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0108 19:00:45.576810 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0108 19:00:46.887962 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0108 19:00:46.960369 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0108 19:00:47.933168 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. I0108 19:00:48.716640 24709 meta_cache.cc:1038] Marking tablet server c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) as failed. W0108 19:00:48.716668 24709 meta_cache.cc:234] Tablet 50603d14716f4c4bb514a91e103fbf64: Replica c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111) W0108 19:00:49.369159 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0108 19:00:49.379686 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0108 19:00:50.621037 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0108 19:00:51.923316 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0108 19:00:52.028177 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0108 19:00:53.140681 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0108 19:00:54.335789 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0108 19:00:54.449352 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0108 19:00:55.654806 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0108 19:00:56.587651 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. W0108 19:00:56.903854 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0108 19:00:58.112578 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. W0108 19:00:58.947804 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0108 19:00:59.468520 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. I0108 19:01:00.522513 24709 linked_list-test-util.h:876] Done collecting results (13799750 rows in 36385ms) W0108 19:01:00.860569 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0108 19:01:01.054785 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0108 19:01:01.902148 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. I0108 19:01:02.197333 24709 linked_list-test-util.h:667] Snapshot: P: 1578538161018821 usec, L: 1 verified. Result: OK I0108 19:01:02.197882 24709 linked_list-test-util.h:620] Verifying Snapshot: P: 1578538461013192 usec, L: 1 Expected Rows: 19548500 I0108 19:01:02.199144 24709 linked_list-test-util.h:639] Calling callback on tserver 61b86269d42a403f94c0a049429fa247 W0108 19:01:03.417757 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0108 19:01:03.458827 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0108 19:01:04.512898 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0108 19:01:05.882306 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0108 19:01:05.968211 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0108 19:01:06.969480 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0108 19:01:08.419075 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0108 19:01:08.458649 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0108 19:01:09.661914 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0108 19:01:10.797185 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0108 19:01:10.881512 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0108 19:01:12.319274 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0108 19:01:13.349748 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0108 19:01:13.545367 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0108 19:01:14.622944 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0108 19:01:15.689091 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0108 19:01:15.886265 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0108 19:01:17.208693 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0108 19:01:18.151515 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0108 19:01:18.513706 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0108 19:01:19.802156 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0108 19:01:20.919143 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0108 19:01:20.972858 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0108 19:01:21.935319 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0108 19:01:23.271634 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0108 19:01:23.675810 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0108 19:01:24.504973 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0108 19:01:25.589354 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0108 19:01:26.114356 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0108 19:01:26.956176 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0108 19:01:28.278581 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0108 19:01:28.748800 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0108 19:01:29.618821 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0108 19:01:30.494932 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0108 19:01:31.264240 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0108 19:01:31.956943 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0108 19:01:33.078963 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0108 19:01:33.447188 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0108 19:01:34.444965 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0108 19:01:35.406687 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0108 19:01:36.283161 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0108 19:01:36.806644 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0108 19:01:37.910179 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. I0108 19:01:38.563228 24709 linked_list-test-util.h:876] Done collecting results (19548500 rows in 36364.1ms) W0108 19:01:38.666579 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0108 19:01:39.287817 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0108 19:01:40.442786 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. I0108 19:01:41.001622 24709 linked_list-test-util.h:667] Snapshot: P: 1578538461013192 usec, L: 1 verified. Result: OK I0108 19:01:41.002228 24709 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 19548500 W0108 19:01:41.039522 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0108 19:01:41.788725 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0108 19:01:43.153637 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0108 19:01:43.300997 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0108 19:01:44.621636 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. I0108 19:01:45.467217 24736 maintenance_manager.cc:326] P 86570b464d814d0eb2725157358e3a2f: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033357 I0108 19:01:45.468089 24735 maintenance_manager.cc:527] P 86570b464d814d0eb2725157358e3a2f: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"bytes_written":441,"delete_count":0,"lbm_write_time_us":38,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":33,"threads_started":1,"update_count":1,"wal-append.queue_time_us":58} W0108 19:01:45.612072 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0108 19:01:45.719254 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0108 19:01:47.127871 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0108 19:01:48.117455 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0108 19:01:48.374341 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0108 19:01:49.567468 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0108 19:01:50.581113 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0108 19:01:50.763420 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0108 19:01:51.959314 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0108 19:01:53.034970 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0108 19:01:53.358144 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0108 19:01:54.562652 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0108 19:01:55.339334 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0108 19:01:55.557989 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0108 19:01:57.035044 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0108 19:01:57.942764 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0108 19:01:57.968931 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0108 19:01:59.485952 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0108 19:02:00.548833 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0108 19:02:00.937486 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0108 19:02:01.851198 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0108 19:02:03.033131 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0108 19:02:03.644703 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0108 19:02:04.350291 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0108 19:02:05.385139 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0108 19:02:06.138155 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0108 19:02:06.570559 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0108 19:02:07.685488 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0108 19:02:08.724711 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0108 19:02:08.833091 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0108 19:02:10.005925 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0108 19:02:11.460074 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0108 19:02:11.464588 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0108 19:02:12.380517 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry. W0108 19:02:13.708166 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0108 19:02:13.990344 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0108 19:02:14.929442 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry. I0108 19:02:16.100653 24709 linked_list-test-util.h:876] Done collecting results (19548500 rows in 34667ms) W0108 19:02:16.269068 30482 consensus_peers.cc:473] T fdf8e522824640c3ab9385967c93743a P 53135810f2d84759ac110009de750c3d -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry. W0108 19:02:16.572721 25049 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry. W0108 19:02:17.524406 25049 consensus_peers.cc:473] T dbe8c3d61048482fb282021487b845a6 P 61b86269d42a403f94c0a049429fa247 -> Peer c6ad452ff3b64d949298a50527e1fbb0 (127.24.33.65:55934): Couldn't send request to peer c6ad452ff3b64d949298a50527e1fbb0. Status: Network error: Client connection negotiation failed: client connection to 127.24.33.65:55934: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry. I0108 19:02:18.514427 24709 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0108 19:02:18.514505 24709 linked_list-test-util.h:801] Successfully verified 19548500 rows I0108 19:02:18.514526 24709 linked_list-test.cc:215] Done with tserver kill test. I0108 19:02:18.517760 24709 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 30468 W0108 19:02:18.554821 25050 connection.cc:655] client connection to 127.24.33.66:33490 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) I0108 19:02:18.554834 24709 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 25040 W0108 19:02:18.554857 25050 consensus_peers.cc:473] T 50603d14716f4c4bb514a91e103fbf64 P 61b86269d42a403f94c0a049429fa247 -> Peer 53135810f2d84759ac110009de750c3d (127.24.33.66:33490): Couldn't send request to peer 53135810f2d84759ac110009de750c3d. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I0108 19:02:18.593457 24709 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.24.33.65:55934 --local_ip_for_outbound_sockets=127.24.33.65 --webserver_port=35659 --webserver_interface=127.24.33.65 --tserver_master_addrs=127.24.33.126:55767 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.24.33.84:49727 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 19:02:18.605458 31129 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 19:02:18.605562 31129 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 19:02:18.606353 31129 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 19:02:18.606377 31129 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.24.33.65 I0108 19:02:18.607363 31129 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.24.33.84:49727 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.24.33.65:55934 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.24.33.65 --webserver_port=35659 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.24.33.126:55767 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.31129 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.24.33.65 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 4dca6b02a5e7b2e589646e79bc6e509f27c13c3d build type RELEASE built by jenkins at 08 Jan 2020 18:26:25 PST on quasar-luegrb-1.vpc.cloudera.com I0108 19:02:18.609525 31129 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 19:02:18.610664 31129 hybrid_clock.cc:186] HybridClock initialized: now 1578538938610651 us; error 21 us; skew 500 ppm I0108 19:02:18.610734 31129 fs_manager.cc:263] Metadata directory not provided I0108 19:02:18.610752 31129 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 19:02:18.612933 31129 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s I0108 19:02:18.613061 31129 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 19:02:18.613108 31129 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 19:02:18.615173 31129 fs_manager.cc:434] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s I0108 19:02:18.615366 31129 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal uuid: "c6ad452ff3b64d949298a50527e1fbb0" format_stamp: "Formatted at 2020-01-09 02:39:20 on quasar-luegrb-1.vpc.cloudera.com" I0108 19:02:18.615397 31129 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578537560432831-24709/linked-list-cluster/ts-0/data/data Total live blocks: 66 Total live bytes: 770766661 Total live bytes (after alignment): 770920448 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 19:02:18.646083 31129 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 19:02:18.646298 31129 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0108 19:02:18.647094 31129 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0108 19:02:18.647110 31129 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0108 19:02:18.647416 31151 tablet_bootstrap.cc:439] T 50603d14716f4c4bb514a91e103fbf64 P c6ad452ff3b64d949298a50527e1fbb0: Bootstrap starting. I0108 19:02:18.647642 31129 ts_tablet_manager.cc:415] Registered 3 tablets I0108 19:02:18.652653 31129 rpc_server.cc:211] RPC server started. Bound to: 127.24.33.65:55934 E0108 19:02:18.652782 31129 webserver.cc:443] Webserver: set_ports_option: cannot bind to 127.24.33.65:35659: 98 (Address already in use) W0108 19:02:18.652804 31129 net_util.cc:463] Trying to use lsof to find any processes listening on 0.0.0.0:35659 I0108 19:02:18.652807 31129 net_util.cc:466] $ export PATH=$PATH:/usr/sbin ; lsof -n -i 'TCP:35659' -sTCP:LISTEN ; for pid in $(lsof -F p -n -i 'TCP:35659' -sTCP:LISTEN | grep p | cut -f 2 -dp) ; do while [ $pid -gt 1 ] ; do ps h -fp $pid ; stat=($(