[==========] Running 2 tests from 1 test case. [----------] Global test environment set-up. [----------] 2 tests from LinkedListTest [ RUN ] LinkedListTest.TestLoadAndVerify WARNING: Logging before InitGoogleLogging() is written to STDERR I0108 16:31:59.792970 24342 test_util.cc:229] Using random seed: 589494085 I0108 16:31:59.796928 24342 linked_list-test.cc:100] Linked List Test Configuration: I0108 16:31:59.796941 24342 linked_list-test.cc:101] -------------- I0108 16:31:59.796947 24342 linked_list-test.cc:102] 50 chains I0108 16:31:59.796950 24342 linked_list-test.cc:103] 3 tablets I0108 16:31:59.796953 24342 linked_list-test.cc:104] Mutations on I0108 16:31:59.796957 24342 linked_list-test.cc:105] -------------- I0108 16:31:59.796983 24342 ts_itest-base.cc:112] Starting cluster with: I0108 16:31:59.796988 24342 ts_itest-base.cc:113] -------------- I0108 16:31:59.796993 24342 ts_itest-base.cc:114] 3 tablet servers I0108 16:31:59.796995 24342 ts_itest-base.cc:115] 3 replicas per TS I0108 16:31:59.796998 24342 ts_itest-base.cc:116] -------------- 2020-01-09T00:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2020-01-09T00:31:59Z Disabled control of system clock Could not open connection to daemon I0108 16:31:59.801652 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.197.190:59202 --webserver_interface=127.23.197.190 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:31:59.812470 24352 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:31:59.812570 24352 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:31:59.813495 24352 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W0108 16:31:59.813510 24352 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:31:59.813516 24352 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W0108 16:31:59.813521 24352 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I0108 16:31:59.814733 24352 master_main.cc:96] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.23.197.190:59202 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/info.pb --webserver_interface=127.23.197.190 --webserver_port=0 --tablet_history_max_age_sec=86400 --never_fsync=true --heap_profile_path=/tmp/kudu-master.24352 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:31:59.816718 24352 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:31:59.817898 24352 hybrid_clock.cc:186] HybridClock initialized: now 1578529919817889 us; error 22 us; skew 500 ppm I0108 16:31:59.817971 24352 fs_manager.cc:263] Metadata directory not provided I0108 16:31:59.817991 24352 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:31:59.818044 24352 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 16:31:59.818058 24352 server_base.cc:453] Attempting to create new FS layout instead I0108 16:31:59.818711 24352 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/instance: uuid: "dc13e8f375d24e5a866a0ef1632607ea" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.818845 24352 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/wal/instance: uuid: "dc13e8f375d24e5a866a0ef1632607ea" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.819679 24352 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/data/block_manager_instance: No such file or directory (error 2) I0108 16:31:59.830605 24352 fs_manager.cc:518] Time spent creating directory manager: real 0.012s user 0.002s sys 0.001s I0108 16:31:59.830760 24352 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:31:59.830829 24352 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:31:59.831766 24352 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0108 16:31:59.831791 24352 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/wal uuid: "dc13e8f375d24e5a866a0ef1632607ea" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.831820 24352 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:31:59.890342 24352 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:31:59.892899 24352 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.190:59202 I0108 16:31:59.893092 24352 webserver.cc:382] Webserver started at http://127.23.197.190:56056/ using document root and password file I0108 16:31:59.893218 24352 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/master-0/data/info.pb I0108 16:31:59.893440 24413 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:31:59.893702 24342 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-master as pid 24352 I0108 16:31:59.893894 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.197.129:0 --local_ip_for_outbound_sockets=127.23.197.129 --webserver_interface=127.23.197.129 --webserver_port=0 --tserver_master_addrs=127.23.197.190:59202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0108 16:31:59.895120 24413 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea: Bootstrap starting. I0108 16:31:59.895347 24413 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea: No blocks or log segments found. Creating new log. I0108 16:31:59.895483 24413 log.cc:815] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea: Log is configured to *not* fsync() on all Append() calls I0108 16:31:59.896575 24413 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea: No bootstrap required, opened a new log I0108 16:31:59.897720 24413 raft_consensus.cc:341] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } I0108 16:31:59.897739 24413 raft_consensus.cc:367] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:31:59.897747 24413 raft_consensus.cc:712] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc13e8f375d24e5a866a0ef1632607ea, State: Initialized, Role: FOLLOWER I0108 16:31:59.897812 24413 consensus_queue.cc:229] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } I0108 16:31:59.897832 24413 raft_consensus.cc:381] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0108 16:31:59.897840 24413 raft_consensus.cc:477] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0108 16:31:59.897851 24413 raft_consensus.cc:2949] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 0 FOLLOWER]: Advancing to term 1 I0108 16:31:59.898139 24413 raft_consensus.cc:499] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } I0108 16:31:59.898175 24413 leader_election.cc:307] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc13e8f375d24e5a866a0ef1632607ea; no voters: I0108 16:31:59.898229 24413 leader_election.cc:293] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [CANDIDATE]: Term 1 election: Requested vote from peers I0108 16:31:59.898247 24416 raft_consensus.cc:2696] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 1 FOLLOWER]: Leader election won for term 1 I0108 16:31:59.898435 24416 raft_consensus.cc:672] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [term 1 LEADER]: Becoming Leader. State: Replica: dc13e8f375d24e5a866a0ef1632607ea, State: Running, Role: LEADER I0108 16:31:59.898473 24416 consensus_queue.cc:206] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } I0108 16:31:59.898600 24413 sys_catalog.cc:340] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "dc13e8f375d24e5a866a0ef1632607ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } } I0108 16:31:59.898617 24413 sys_catalog.cc:343] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: This master's current role is: LEADER I0108 16:31:59.898713 24413 sys_catalog.cc:440] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: configured and running, proceeding with master startup. I0108 16:31:59.898850 24417 sys_catalog.cc:340] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc13e8f375d24e5a866a0ef1632607ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } } I0108 16:31:59.899075 24417 sys_catalog.cc:343] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: This master's current role is: LEADER I0108 16:31:59.898880 24418 sys_catalog.cc:340] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc13e8f375d24e5a866a0ef1632607ea. Latest consensus state: current_term: 1 leader_uuid: "dc13e8f375d24e5a866a0ef1632607ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc13e8f375d24e5a866a0ef1632607ea" member_type: VOTER } } I0108 16:31:59.899099 24418 sys_catalog.cc:343] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea [sys.catalog]: This master's current role is: LEADER I0108 16:31:59.898872 24420 catalog_manager.cc:1089] Loading table and tablet metadata into memory... I0108 16:31:59.899317 24420 catalog_manager.cc:1100] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:31:59.904598 24414 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:31:59.904700 24414 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:31:59.905524 24414 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:31:59.905555 24414 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.197.129 I0108 16:31:59.906615 24414 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.197.129:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.23.197.129 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.197.190:59202 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24414 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.197.129 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:31:59.908675 24414 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:31:59.909821 24414 hybrid_clock.cc:186] HybridClock initialized: now 1578529919909806 us; error 19 us; skew 500 ppm I0108 16:31:59.909907 24414 fs_manager.cc:263] Metadata directory not provided I0108 16:31:59.909929 24414 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:31:59.909986 24414 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 16:31:59.910002 24414 server_base.cc:453] Attempting to create new FS layout instead I0108 16:31:59.910682 24414 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/instance: uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.910818 24414 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/instance: uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.911650 24414 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/data/block_manager_instance: No such file or directory (error 2) I0108 16:31:59.914815 24414 fs_manager.cc:518] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s I0108 16:31:59.914958 24414 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:31:59.915027 24414 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:31:59.915962 24414 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0108 16:31:59.915988 24414 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.916015 24414 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:31:59.936489 24414 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:31:59.936692 24414 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0108 16:31:59.936705 24414 ts_tablet_manager.cc:415] Registered 0 tablets I0108 16:31:59.942973 24420 catalog_manager.cc:984] Generated new certificate authority record I0108 16:31:59.943099 24414 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.129:35094 I0108 16:31:59.943343 24414 webserver.cc:382] Webserver started at http://127.23.197.129:37729/ using document root and password file I0108 16:31:59.943486 24414 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/info.pb I0108 16:31:59.943931 24420 catalog_manager.cc:1109] Loading token signing keys... I0108 16:31:59.945659 24342 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24414 I0108 16:31:59.945873 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.197.130:0 --local_ip_for_outbound_sockets=127.23.197.130 --webserver_interface=127.23.197.130 --webserver_port=0 --tserver_master_addrs=127.23.197.190:59202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} I0108 16:31:59.947044 24545 heartbeater.cc:322] Connected to a master server at 127.23.197.190:59202 I0108 16:31:59.947060 24545 heartbeater.cc:406] Registering TS with master... I0108 16:31:59.947099 24545 heartbeater.cc:455] Master 127.23.197.190:59202 requested a full tablet report, sending... I0108 16:31:59.947309 24380 ts_manager.cc:192] Registered new tserver with Master: e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094) I0108 16:31:59.951414 24420 catalog_manager.cc:4455] T 00000000000000000000000000000000 P dc13e8f375d24e5a866a0ef1632607ea: Generated new TSK 0 I0108 16:31:59.951462 24420 catalog_manager.cc:1119] Initializing in-progress tserver states... WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:31:59.956907 24549 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:31:59.957011 24549 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:31:59.957813 24549 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:31:59.957844 24549 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.197.130 I0108 16:31:59.958873 24549 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.197.130:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/info.pb --webserver_interface=127.23.197.130 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.197.190:59202 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24549 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.197.130 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:31:59.960960 24549 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:31:59.962128 24549 hybrid_clock.cc:186] HybridClock initialized: now 1578529919962118 us; error 19 us; skew 500 ppm I0108 16:31:59.962219 24549 fs_manager.cc:263] Metadata directory not provided I0108 16:31:59.962242 24549 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:31:59.962293 24549 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 16:31:59.962308 24549 server_base.cc:453] Attempting to create new FS layout instead I0108 16:31:59.962950 24549 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/instance: uuid: "02a5a9f6601045b6b339c43c526fc29e" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.963086 24549 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/instance: uuid: "02a5a9f6601045b6b339c43c526fc29e" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.963915 24549 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/data/block_manager_instance: No such file or directory (error 2) I0108 16:31:59.966662 24549 fs_manager.cc:518] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s I0108 16:31:59.966802 24549 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:31:59.966869 24549 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:31:59.967780 24549 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I0108 16:31:59.967806 24549 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal uuid: "02a5a9f6601045b6b339c43c526fc29e" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:31:59.967833 24549 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:31:59.996615 24549 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:31:59.996819 24549 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0108 16:31:59.996832 24549 ts_tablet_manager.cc:415] Registered 0 tablets I0108 16:32:00.002518 24549 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.130:37520 I0108 16:32:00.002753 24549 webserver.cc:382] Webserver started at http://127.23.197.130:44676/ using document root and password file I0108 16:32:00.002921 24549 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/info.pb I0108 16:32:00.006232 24670 heartbeater.cc:322] Connected to a master server at 127.23.197.190:59202 I0108 16:32:00.006255 24670 heartbeater.cc:406] Registering TS with master... I0108 16:32:00.006297 24670 heartbeater.cc:455] Master 127.23.197.190:59202 requested a full tablet report, sending... I0108 16:32:00.006472 24380 ts_manager.cc:192] Registered new tserver with Master: 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) I0108 16:32:00.007238 24380 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.197.130:51894 I0108 16:32:00.008380 24342 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24549 I0108 16:32:00.008587 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.197.131:0 --local_ip_for_outbound_sockets=127.23.197.131 --webserver_interface=127.23.197.131 --webserver_port=0 --tserver_master_addrs=127.23.197.190:59202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:32:00.019140 24673 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:32:00.019243 24673 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:32:00.020064 24673 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:32:00.020094 24673 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.197.131 I0108 16:32:00.021132 24673 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.197.131:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/info.pb --webserver_interface=127.23.197.131 --webserver_port=0 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.197.190:59202 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.24673 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.197.131 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:32:00.023197 24673 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:32:00.024341 24673 hybrid_clock.cc:186] HybridClock initialized: now 1578529920024331 us; error 19 us; skew 500 ppm I0108 16:32:00.024442 24673 fs_manager.cc:263] Metadata directory not provided I0108 16:32:00.024472 24673 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:32:00.024526 24673 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I0108 16:32:00.024540 24673 server_base.cc:453] Attempting to create new FS layout instead I0108 16:32:00.025233 24673 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/instance: uuid: "72e6f176048945dea46e0f4305315d3c" format_stamp: "Formatted at 2020-01-09 00:32:00 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:32:00.025374 24673 fs_manager.cc:617] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/instance: uuid: "72e6f176048945dea46e0f4305315d3c" format_stamp: "Formatted at 2020-01-09 00:32:00 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:32:00.026221 24673 dir_util.cc:225] Instance is unhealthy: Not found: Failed to read metadata file from /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/data/block_manager_instance: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/data/block_manager_instance: No such file or directory (error 2) I0108 16:32:00.029129 24673 fs_manager.cc:518] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s I0108 16:32:00.029273 24673 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:32:00.029345 24673 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:32:00.030274 24673 fs_manager.cc:434] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s I0108 16:32:00.030300 24673 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal uuid: "72e6f176048945dea46e0f4305315d3c" format_stamp: "Formatted at 2020-01-09 00:32:00 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:32:00.030329 24673 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:32:00.043905 24673 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:32:00.044107 24673 ts_tablet_manager.cc:395] Loaded tablet metadata (0 total tablets, 0 live tablets) I0108 16:32:00.044121 24673 ts_tablet_manager.cc:415] Registered 0 tablets I0108 16:32:00.049371 24673 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.131:49342 I0108 16:32:00.049585 24673 webserver.cc:382] Webserver started at http://127.23.197.131:52477/ using document root and password file I0108 16:32:00.049719 24673 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/info.pb I0108 16:32:00.050246 24342 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 24673 I0108 16:32:00.052781 24794 heartbeater.cc:322] Connected to a master server at 127.23.197.190:59202 I0108 16:32:00.052804 24794 heartbeater.cc:406] Registering TS with master... I0108 16:32:00.052841 24794 heartbeater.cc:455] Master 127.23.197.190:59202 requested a full tablet report, sending... I0108 16:32:00.052975 24380 ts_manager.cc:192] Registered new tserver with Master: 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:32:00.053637 24380 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.197.131:58617 I0108 16:32:00.053700 24342 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0108 16:32:00.058542 24380 catalog_manager.cc:1454] Servicing CreateTable request from {username='jenkins'} at 127.23.197.190:42457: name: "TestTable" schema { columns { name: "rand_key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "link_to" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "insert_ts" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "updated" type: BOOL is_key: false is_nullable: false read_default_value: "\000" encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { rows: "\004\001\252\252\252\252\252\252\252*\004\001TUUUUUUU" } partition_schema { range_schema { columns { name: "rand_key" } } } W0108 16:32:00.058619 24380 catalog_manager.cc:1619] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers. I0108 16:32:00.061946 24481 tablet_service.cc:1046] Processing CreateTablet for tablet 2c3d9174c33442d28f28f4d0cda9fc51 (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0108 16:32:00.062029 24481 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2c3d9174c33442d28f28f4d0cda9fc51. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.062460 24480 tablet_service.cc:1046] Processing CreateTablet for tablet c0c87b04bc0f4f37876458fbfb4bdf03 (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0108 16:32:00.062507 24480 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0c87b04bc0f4f37876458fbfb4bdf03. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.063516 24605 tablet_service.cc:1046] Processing CreateTablet for tablet 97a6791627c941a48f3269f2d00d1eea (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0108 16:32:00.063534 24606 tablet_service.cc:1046] Processing CreateTablet for tablet c0c87b04bc0f4f37876458fbfb4bdf03 (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0108 16:32:00.063581 24605 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 97a6791627c941a48f3269f2d00d1eea. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.063585 24479 tablet_service.cc:1046] Processing CreateTablet for tablet 97a6791627c941a48f3269f2d00d1eea (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0108 16:32:00.063594 24607 tablet_service.cc:1046] Processing CreateTablet for tablet 2c3d9174c33442d28f28f4d0cda9fc51 (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0108 16:32:00.063637 24607 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2c3d9174c33442d28f28f4d0cda9fc51. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.063637 24479 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 97a6791627c941a48f3269f2d00d1eea. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.063691 24606 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0c87b04bc0f4f37876458fbfb4bdf03. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.065500 24811 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Bootstrap starting. I0108 16:32:00.065594 24810 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Bootstrap starting. I0108 16:32:00.065755 24811 tablet_bootstrap.cc:590] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: No blocks or log segments found. Creating new log. I0108 16:32:00.065855 24810 tablet_bootstrap.cc:590] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: No blocks or log segments found. Creating new log. I0108 16:32:00.066006 24810 log.cc:815] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Log is configured to *not* fsync() on all Append() calls I0108 16:32:00.066552 24811 log.cc:815] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Log is configured to *not* fsync() on all Append() calls I0108 16:32:00.066639 24730 tablet_service.cc:1046] Processing CreateTablet for tablet c0c87b04bc0f4f37876458fbfb4bdf03 (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION 3074457345618258602 <= VALUES < 6148914691236517204 I0108 16:32:00.066639 24731 tablet_service.cc:1046] Processing CreateTablet for tablet 2c3d9174c33442d28f28f4d0cda9fc51 (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION VALUES < 3074457345618258602 I0108 16:32:00.066730 24730 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet c0c87b04bc0f4f37876458fbfb4bdf03. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.066850 24810 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: No bootstrap required, opened a new log I0108 16:32:00.066876 24810 ts_tablet_manager.cc:1105] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0108 16:32:00.067046 24811 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: No bootstrap required, opened a new log I0108 16:32:00.067064 24811 ts_tablet_manager.cc:1105] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0108 16:32:00.067986 24811 raft_consensus.cc:341] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.067998 24811 raft_consensus.cc:367] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.068003 24811 raft_consensus.cc:712] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a5a9f6601045b6b339c43c526fc29e, State: Initialized, Role: FOLLOWER I0108 16:32:00.068059 24811 consensus_queue.cc:229] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.068161 24811 ts_tablet_manager.cc:1133] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0108 16:32:00.068176 24811 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Bootstrap starting. I0108 16:32:00.068205 24810 raft_consensus.cc:341] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.068219 24810 raft_consensus.cc:367] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.068226 24810 raft_consensus.cc:712] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61264bf360a4a62a3a5e3b9f2875f5b, State: Initialized, Role: FOLLOWER I0108 16:32:00.068287 24810 consensus_queue.cc:229] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.068348 24811 tablet_bootstrap.cc:590] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: No blocks or log segments found. Creating new log. I0108 16:32:00.068406 24810 ts_tablet_manager.cc:1133] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0108 16:32:00.068428 24810 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Bootstrap starting. I0108 16:32:00.068523 24670 heartbeater.cc:447] Master 127.23.197.190:59202 was elected leader, sending a full tablet report... I0108 16:32:00.068611 24729 tablet_service.cc:1046] Processing CreateTablet for tablet 97a6791627c941a48f3269f2d00d1eea (table=TestTable [id=04df3e3970944b7a9c5b1b40fcfebb0b]), partition=RANGE (rand_key) PARTITION VALUES >= 6148914691236517204 I0108 16:32:00.068660 24810 tablet_bootstrap.cc:590] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: No blocks or log segments found. Creating new log. I0108 16:32:00.068771 24729 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 97a6791627c941a48f3269f2d00d1eea. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.068841 24731 data_dirs.cc:394] Could only allocate 1 dirs of requested 3 for tablet 2c3d9174c33442d28f28f4d0cda9fc51. 1 dirs total, 0 dirs full, 0 dirs failed I0108 16:32:00.069031 24811 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: No bootstrap required, opened a new log I0108 16:32:00.069048 24811 ts_tablet_manager.cc:1105] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0108 16:32:00.069142 24811 raft_consensus.cc:341] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.069150 24811 raft_consensus.cc:367] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.069155 24811 raft_consensus.cc:712] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a5a9f6601045b6b339c43c526fc29e, State: Initialized, Role: FOLLOWER I0108 16:32:00.069191 24811 consensus_queue.cc:229] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.069221 24811 ts_tablet_manager.cc:1133] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 16:32:00.069234 24811 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Bootstrap starting. I0108 16:32:00.069447 24811 tablet_bootstrap.cc:590] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: No blocks or log segments found. Creating new log. I0108 16:32:00.069480 24380 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.197.129:42384 I0108 16:32:00.070140 24817 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Bootstrap starting. I0108 16:32:00.070173 24810 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: No bootstrap required, opened a new log I0108 16:32:00.070191 24810 ts_tablet_manager.cc:1105] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0108 16:32:00.070292 24810 raft_consensus.cc:341] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.070308 24810 raft_consensus.cc:367] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.070315 24810 raft_consensus.cc:712] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61264bf360a4a62a3a5e3b9f2875f5b, State: Initialized, Role: FOLLOWER I0108 16:32:00.070353 24810 consensus_queue.cc:229] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.070359 24811 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: No bootstrap required, opened a new log I0108 16:32:00.070391 24810 ts_tablet_manager.cc:1133] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 16:32:00.070399 24811 ts_tablet_manager.cc:1105] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I0108 16:32:00.070446 24810 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Bootstrap starting. I0108 16:32:00.070402 24545 heartbeater.cc:447] Master 127.23.197.190:59202 was elected leader, sending a full tablet report... I0108 16:32:00.070518 24811 raft_consensus.cc:341] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.070535 24811 raft_consensus.cc:367] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.070549 24811 raft_consensus.cc:712] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a5a9f6601045b6b339c43c526fc29e, State: Initialized, Role: FOLLOWER I0108 16:32:00.070586 24811 consensus_queue.cc:229] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.070633 24810 tablet_bootstrap.cc:590] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: No blocks or log segments found. Creating new log. I0108 16:32:00.070670 24811 ts_tablet_manager.cc:1133] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I0108 16:32:00.070832 24817 tablet_bootstrap.cc:590] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: No blocks or log segments found. Creating new log. I0108 16:32:00.071027 24817 log.cc:815] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Log is configured to *not* fsync() on all Append() calls I0108 16:32:00.071141 24810 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: No bootstrap required, opened a new log I0108 16:32:00.071161 24810 ts_tablet_manager.cc:1105] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I0108 16:32:00.071230 24810 raft_consensus.cc:341] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.071241 24810 raft_consensus.cc:367] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.071244 24810 raft_consensus.cc:712] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61264bf360a4a62a3a5e3b9f2875f5b, State: Initialized, Role: FOLLOWER I0108 16:32:00.071269 24810 consensus_queue.cc:229] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.071316 24810 ts_tablet_manager.cc:1133] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 16:32:00.071664 24817 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: No bootstrap required, opened a new log I0108 16:32:00.071691 24817 ts_tablet_manager.cc:1105] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I0108 16:32:00.072886 24817 raft_consensus.cc:341] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.072906 24817 raft_consensus.cc:367] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.072916 24817 raft_consensus.cc:712] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Initialized, Role: FOLLOWER I0108 16:32:00.072968 24817 consensus_queue.cc:229] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.073089 24817 ts_tablet_manager.cc:1133] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0108 16:32:00.073091 24794 heartbeater.cc:447] Master 127.23.197.190:59202 was elected leader, sending a full tablet report... I0108 16:32:00.073117 24817 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Bootstrap starting. I0108 16:32:00.073335 24817 tablet_bootstrap.cc:590] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: No blocks or log segments found. Creating new log. I0108 16:32:00.073819 24817 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: No bootstrap required, opened a new log I0108 16:32:00.073843 24817 ts_tablet_manager.cc:1105] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I0108 16:32:00.073940 24817 raft_consensus.cc:341] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.073956 24817 raft_consensus.cc:367] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.073961 24817 raft_consensus.cc:712] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Initialized, Role: FOLLOWER I0108 16:32:00.074002 24817 consensus_queue.cc:229] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.074045 24817 ts_tablet_manager.cc:1133] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 16:32:00.074064 24817 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Bootstrap starting. I0108 16:32:00.074249 24817 tablet_bootstrap.cc:590] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: No blocks or log segments found. Creating new log. I0108 16:32:00.074784 24817 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: No bootstrap required, opened a new log I0108 16:32:00.074806 24817 ts_tablet_manager.cc:1105] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.002s I0108 16:32:00.074890 24817 raft_consensus.cc:341] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.074905 24817 raft_consensus.cc:367] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0108 16:32:00.074911 24817 raft_consensus.cc:712] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Initialized, Role: FOLLOWER I0108 16:32:00.074947 24817 consensus_queue.cc:229] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.075057 24817 ts_tablet_manager.cc:1133] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 16:32:00.096419 24814 raft_consensus.cc:477] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 16:32:00.096519 24814 raft_consensus.cc:499] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.096666 24814 leader_election.cc:293] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:32:00.098760 24751 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "2c3d9174c33442d28f28f4d0cda9fc51" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" is_pre_election: true I0108 16:32:00.098788 24751 raft_consensus.cc:2396] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a5a9f6601045b6b339c43c526fc29e in term 0. I0108 16:32:00.098803 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "2c3d9174c33442d28f28f4d0cda9fc51" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" is_pre_election: true I0108 16:32:00.098824 24501 raft_consensus.cc:2396] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a5a9f6601045b6b339c43c526fc29e in term 0. I0108 16:32:00.098917 24558 leader_election.cc:307] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a5a9f6601045b6b339c43c526fc29e, 72e6f176048945dea46e0f4305315d3c; no voters: I0108 16:32:00.098978 24814 raft_consensus.cc:2696] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0108 16:32:00.098999 24814 raft_consensus.cc:477] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0108 16:32:00.099007 24814 raft_consensus.cc:2949] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.099603 24814 raft_consensus.cc:499] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.099642 24814 leader_election.cc:293] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 1 election: Requested vote from peers e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:32:00.099753 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "2c3d9174c33442d28f28f4d0cda9fc51" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" I0108 16:32:00.099753 24751 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "2c3d9174c33442d28f28f4d0cda9fc51" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" I0108 16:32:00.099769 24751 raft_consensus.cc:2949] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.099771 24501 raft_consensus.cc:2949] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.100618 24501 raft_consensus.cc:2396] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a5a9f6601045b6b339c43c526fc29e in term 1. I0108 16:32:00.100682 24751 raft_consensus.cc:2396] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a5a9f6601045b6b339c43c526fc29e in term 1. I0108 16:32:00.100692 24558 leader_election.cc:307] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a5a9f6601045b6b339c43c526fc29e, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: I0108 16:32:00.100720 24814 raft_consensus.cc:2696] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Leader election won for term 1 I0108 16:32:00.100824 24814 raft_consensus.cc:672] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [term 1 LEADER]: Becoming Leader. State: Replica: 02a5a9f6601045b6b339c43c526fc29e, State: Running, Role: LEADER I0108 16:32:00.100879 24814 consensus_queue.cc:206] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.101259 24380 catalog_manager.cc:4143] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e reported cstate change: term changed from 0 to 1, leader changed from to 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130). New cstate: current_term: 1 leader_uuid: "02a5a9f6601045b6b339c43c526fc29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } health_report { overall_health: UNKNOWN } } } W0108 16:32:00.193953 24546 tablet.cc:1792] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Can't schedule compaction. Clean time has not been advanced past its initial value. W0108 16:32:00.253262 24671 tablet.cc:1792] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Can't schedule compaction. Clean time has not been advanced past its initial value. W0108 16:32:00.300061 24795 tablet.cc:1792] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0108 16:32:00.323956 24822 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 16:32:00.324024 24822 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.324142 24822 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094) I0108 16:32:00.326246 24627 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "72e6f176048945dea46e0f4305315d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a5a9f6601045b6b339c43c526fc29e" is_pre_election: true I0108 16:32:00.326256 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "72e6f176048945dea46e0f4305315d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" is_pre_election: true I0108 16:32:00.326277 24501 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e6f176048945dea46e0f4305315d3c in term 0. I0108 16:32:00.326277 24627 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e6f176048945dea46e0f4305315d3c in term 0. I0108 16:32:00.326362 24682 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: I0108 16:32:00.326412 24822 raft_consensus.cc:2696] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0108 16:32:00.326426 24822 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0108 16:32:00.326432 24822 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.328433 24815 raft_consensus.cc:477] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0108 16:32:00.328480 24815 raft_consensus.cc:499] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.328579 24815 leader_election.cc:293] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:32:00.330462 24751 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "c0c87b04bc0f4f37876458fbfb4bdf03" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" is_pre_election: true I0108 16:32:00.330487 24751 raft_consensus.cc:2396] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61264bf360a4a62a3a5e3b9f2875f5b in term 0. I0108 16:32:00.330591 24432 leader_election.cc:307] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: I0108 16:32:00.330647 24815 raft_consensus.cc:2696] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Leader pre-election won for term 1 I0108 16:32:00.330660 24815 raft_consensus.cc:477] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0108 16:32:00.330664 24815 raft_consensus.cc:2949] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.330660 24627 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "c0c87b04bc0f4f37876458fbfb4bdf03" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a5a9f6601045b6b339c43c526fc29e" is_pre_election: true I0108 16:32:00.330678 24627 raft_consensus.cc:2396] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61264bf360a4a62a3a5e3b9f2875f5b in term 0. I0108 16:32:00.339025 24815 raft_consensus.cc:499] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.339025 24822 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.339078 24815 leader_election.cc:293] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 1 election: Requested vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:32:00.339078 24822 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 1 election: Requested vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094) I0108 16:32:00.339170 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "72e6f176048945dea46e0f4305315d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" I0108 16:32:00.339179 24627 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "c0c87b04bc0f4f37876458fbfb4bdf03" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a5a9f6601045b6b339c43c526fc29e" I0108 16:32:00.339188 24501 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.339196 24627 raft_consensus.cc:2949] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.339223 24751 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "c0c87b04bc0f4f37876458fbfb4bdf03" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" I0108 16:32:00.339236 24751 raft_consensus.cc:2949] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.339294 24626 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "72e6f176048945dea46e0f4305315d3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a5a9f6601045b6b339c43c526fc29e" I0108 16:32:00.339313 24626 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 0 FOLLOWER]: Advancing to term 1 I0108 16:32:00.340273 24501 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e6f176048945dea46e0f4305315d3c in term 1. I0108 16:32:00.340293 24751 raft_consensus.cc:2396] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61264bf360a4a62a3a5e3b9f2875f5b in term 1. I0108 16:32:00.340315 24627 raft_consensus.cc:2396] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61264bf360a4a62a3a5e3b9f2875f5b in term 1. I0108 16:32:00.340349 24682 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: I0108 16:32:00.340389 24822 raft_consensus.cc:2696] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Leader election won for term 1 I0108 16:32:00.340404 24432 leader_election.cc:307] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: I0108 16:32:00.340445 24815 raft_consensus.cc:2696] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Leader election won for term 1 I0108 16:32:00.340477 24822 raft_consensus.cc:672] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 LEADER]: Becoming Leader. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Running, Role: LEADER I0108 16:32:00.340514 24815 raft_consensus.cc:672] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 LEADER]: Becoming Leader. State: Replica: e61264bf360a4a62a3a5e3b9f2875f5b, State: Running, Role: LEADER I0108 16:32:00.340528 24822 consensus_queue.cc:206] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.340541 24626 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e6f176048945dea46e0f4305315d3c in term 1. I0108 16:32:00.340557 24815 consensus_queue.cc:206] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:32:00.340904 24380 catalog_manager.cc:4143] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c reported cstate change: term changed from 0 to 1, leader changed from to 72e6f176048945dea46e0f4305315d3c (127.23.197.131). New cstate: current_term: 1 leader_uuid: "72e6f176048945dea46e0f4305315d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } health_report { overall_health: HEALTHY } } } I0108 16:32:00.340904 24379 catalog_manager.cc:4143] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b reported cstate change: term changed from 0 to 1, leader changed from to e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129). New cstate: current_term: 1 leader_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } health_report { overall_health: UNKNOWN } } } I0108 16:32:00.343634 24342 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0108 16:32:00.343876 24342 ts_itest-base.cc:233] Waiting for 3 tablets on tserver e61264bf360a4a62a3a5e3b9f2875f5b to finish bootstrapping I0108 16:32:00.345376 24342 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 02a5a9f6601045b6b339c43c526fc29e to finish bootstrapping I0108 16:32:00.346835 24342 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 72e6f176048945dea46e0f4305315d3c to finish bootstrapping I0108 16:32:00.348878 24342 linked_list-test-util.h:512] Running for: 900.000s I0108 16:32:00.348898 24342 linked_list-test-util.h:513] Sampling every 300000000 us I0108 16:32:00.348908 24342 linked_list-test-util.h:519] Written 0 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 0 Mean: 0 Percentiles: 0% (min) = 0 25% = 0 50% (med) = 0 75% = 0 95% = 0 99% = 0 99.9% = 0 99.99% = 0 100% (max) = 0 I0108 16:32:00.349388 24835 linked_list-test-util.h:379] Curl thread will poll the following URLs every 1000 ms: I0108 16:32:00.349653 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/metrics I0108 16:32:00.349731 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/tablets I0108 16:32:00.349808 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/maintenance-manager I0108 16:32:00.349885 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/stacks I0108 16:32:00.349961 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/tables I0108 16:32:00.350075 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/threadz?group=all I0108 16:32:00.350162 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/stacks I0108 16:32:00.350208 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/threadz?group=all I0108 16:32:00.350244 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/stacks I0108 16:32:00.350282 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/tablets I0108 16:32:00.350320 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/scans I0108 16:32:00.350358 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/mem-trackers I0108 16:32:00.350397 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/transactions?tablet_id=c0c87b04bc0f4f37876458fbfb4bdf03 I0108 16:32:00.350437 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/scans I0108 16:32:00.350492 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/threadz?group=all I0108 16:32:00.350533 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/dump-entities I0108 16:32:00.350571 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/maintenance-manager I0108 16:32:00.350579 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/maintenance-manager I0108 16:32:00.350582 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/tablets I0108 16:32:00.350586 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/masters I0108 16:32:00.350591 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/metrics I0108 16:32:00.350594 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/threadz I0108 16:32:00.350597 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/metrics I0108 16:32:00.350600 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/scans I0108 16:32:00.350603 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/tablet-servers I0108 16:32:00.350606 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/mem-trackers I0108 16:32:00.350610 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/mem-trackers I0108 16:32:00.350612 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/stacks I0108 16:32:00.350615 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/transactions?tablet_id=c0c87b04bc0f4f37876458fbfb4bdf03 I0108 16:32:00.350620 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/transactions?tablet_id=c0c87b04bc0f4f37876458fbfb4bdf03 I0108 16:32:00.350622 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/threadz?group=all I0108 16:32:00.350625 24835 linked_list-test-util.h:382] http://127.23.197.190:56056/threadz I0108 16:32:00.350628 24835 linked_list-test-util.h:382] http://127.23.197.130:44676/mem-trackers I0108 16:32:00.350632 24835 linked_list-test-util.h:382] http://127.23.197.131:52477/threadz I0108 16:32:00.350636 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/metrics I0108 16:32:00.350644 24835 linked_list-test-util.h:382] http://127.23.197.129:37729/threadz I0108 16:32:00.353497 24501 raft_consensus.cc:1195] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Refusing update from remote peer 02a5a9f6601045b6b339c43c526fc29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 16:32:00.353516 24751 raft_consensus.cc:1195] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Refusing update from remote peer 02a5a9f6601045b6b339c43c526fc29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 16:32:00.353652 24826 consensus_queue.cc:982] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 16:32:00.353708 24826 consensus_queue.cc:982] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 16:32:00.355283 24841 mvcc.cc:203] Tried to move safe_time back from 6465658553766973440 to 6465658552733384704. Current Snapshot: MvccSnapshot[committed={T|T < 6465658553766973440}] I0108 16:32:00.355592 24501 raft_consensus.cc:1195] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Refusing update from remote peer 72e6f176048945dea46e0f4305315d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 16:32:00.355650 24626 raft_consensus.cc:1195] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Refusing update from remote peer 72e6f176048945dea46e0f4305315d3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 16:32:00.355747 24830 consensus_queue.cc:982] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 16:32:00.355799 24822 consensus_queue.cc:982] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 16:32:00.355952 24751 raft_consensus.cc:1195] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Refusing update from remote peer e61264bf360a4a62a3a5e3b9f2875f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 16:32:00.355994 24627 raft_consensus.cc:1195] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Refusing update from remote peer e61264bf360a4a62a3a5e3b9f2875f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0108 16:32:00.356011 24844 mvcc.cc:203] Tried to move safe_time back from 6465658553766973440 to 6465658552733384704. Current Snapshot: MvccSnapshot[committed={T|T < 6465658553766973440}] I0108 16:32:00.356173 24842 mvcc.cc:203] Tried to move safe_time back from 6465658553766973440 to 6465658552733384704. Current Snapshot: MvccSnapshot[committed={T|T < 6465658553766973440}] I0108 16:32:00.356199 24815 consensus_queue.cc:982] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0108 16:32:00.356287 24831 consensus_queue.cc:982] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 10000 Mean: 1489.79 Percentiles: 0% (min) = 781 25% = 1177 50% (med) = 1309 75% = 1481 95% = 1856 99% = 4920 99.9% = 35392 99.99% = 50976 100% (max) = 57354 I0108 16:32:17.135989 24342 linked_list-test-util.h:519] Written 500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 20000 Mean: 1526.25 Percentiles: 0% (min) = 767 25% = 1201 50% (med) = 1337 75% = 1512 95% = 1924 99% = 5448 99.9% = 34624 99.99% = 48288 100% (max) = 57354 I0108 16:32:34.567785 24342 linked_list-test-util.h:519] Written 1000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 30000 Mean: 1538.68 Percentiles: 0% (min) = 728 25% = 1210 50% (med) = 1345 75% = 1518 95% = 1922 99% = 5604 99.9% = 35392 99.99% = 48704 100% (max) = 57354 I0108 16:32:52.168884 24342 linked_list-test-util.h:519] Written 1500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 40000 Mean: 1548.8 Percentiles: 0% (min) = 728 25% = 1222 50% (med) = 1361 75% = 1534 95% = 1935 99% = 5600 99.9% = 34624 99.99% = 48704 100% (max) = 57354 I0108 16:33:09.865923 24342 linked_list-test-util.h:519] Written 2000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 50000 Mean: 1560.74 Percentiles: 0% (min) = 728 25% = 1232 50% (med) = 1374 75% = 1551 95% = 1952 99% = 5640 99.9% = 34848 99.99% = 48288 100% (max) = 57354 I0108 16:33:27.866555 24342 linked_list-test-util.h:519] Written 2500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 60000 Mean: 1568.51 Percentiles: 0% (min) = 728 25% = 1239 50% (med) = 1381 75% = 1559 95% = 1963 99% = 5756 99.9% = 34208 99.99% = 48288 100% (max) = 59583 I0108 16:33:45.845892 24342 linked_list-test-util.h:519] Written 3000000 rows in chain I0108 16:33:59.993106 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033360 I0108 16:33:59.999945 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.003s sys 0.000s Metrics: {"bytes_written":7608,"cfile_init":1,"dirs.queue_time_us":77,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":132,"lbm_writes_lt_1ms":27,"rows_written":6,"thread_start_us":65,"threads_started":2,"wal-append.queue_time_us":45} I0108 16:34:00.000027 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 581 bytes on disk I0108 16:34:00.000198 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4} I0108 16:34:00.111673 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033345 I0108 16:34:00.151734 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033355 I0108 16:34:00.304404 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033399 I0108 16:34:04.253085 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 4.100s user 0.854s sys 0.069s Metrics: {"bytes_written":9964992,"cfile_init":1,"compiler_manager_pool.queue_time_us":8623,"compiler_manager_pool.run_cpu_time_us":167749,"compiler_manager_pool.run_wall_time_us":433513,"delete_count":0,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":170637,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":122148,"lbm_writes_1-10_ms":29,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":991,"reinsert_count":0,"rows_written":566455,"thread_start_us":70,"threads_started":2,"update_count":1834} I0108 16:34:04.255125 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41963727 bytes of WAL I0108 16:34:04.257074 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:34:04.257386 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000001 (ops 1-12519) I0108 16:34:04.257489 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000002 (ops 12520-24980) I0108 16:34:04.260856 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000003 (ops 24981-37471) I0108 16:34:04.260929 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000004 (ops 37472-49960) I0108 16:34:04.261059 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000005 (ops 49961-62486) ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 70000 Mean: 1579.66 Percentiles: 0% (min) = 728 25% = 1244 50% (med) = 1389 75% = 1569 95% = 1983 99% = 5744 99.9% = 34432 99.99% = 51712 100% (max) = 80298 I0108 16:34:04.300514 24342 linked_list-test-util.h:519] Written 3500000 rows in chain I0108 16:34:04.317322 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.060s user 0.001s sys 0.012s Metrics: {} I0108 16:34:04.319308 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 11245773 bytes on disk I0108 16:34:04.319489 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0108 16:34:04.319567 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034513 I0108 16:34:04.402873 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 4.290s user 0.907s sys 0.056s Metrics: {"bytes_written":9946798,"cfile_init":1,"compiler_manager_pool.queue_time_us":71,"compiler_manager_pool.run_cpu_time_us":177276,"compiler_manager_pool.run_wall_time_us":841315,"delete_count":0,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":38968,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":140855,"lbm_writes_1-10_ms":44,"lbm_writes_lt_1ms":975,"reinsert_count":0,"rows_written":565570,"thread_start_us":91,"threads_started":2,"update_count":1720} I0108 16:34:04.403275 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41930748 bytes of WAL I0108 16:34:04.405503 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:34:04.406603 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000001 (ops 1-12504) I0108 16:34:04.407040 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000002 (ops 12505-24954) I0108 16:34:04.408290 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000003 (ops 24955-37454) I0108 16:34:04.408398 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000004 (ops 37455-49819) I0108 16:34:04.408459 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000005 (ops 49820-62300) I0108 16:34:04.512887 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.107s user 0.005s sys 0.009s Metrics: {} I0108 16:34:04.513069 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 11243404 bytes on disk I0108 16:34:04.513464 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:34:04.513639 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.034568 I0108 16:34:04.606782 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 4.301s user 0.885s sys 0.085s Metrics: {"bytes_written":9981004,"cfile_init":1,"compiler_manager_pool.queue_time_us":8944,"compiler_manager_pool.run_cpu_time_us":174244,"compiler_manager_pool.run_wall_time_us":793603,"delete_count":0,"dirs.queue_time_us":4228,"dirs.run_cpu_time_us":101,"dirs.run_wall_time_us":30822,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":107816,"lbm_writes_1-10_ms":33,"lbm_writes_lt_1ms":988,"reinsert_count":0,"rows_written":567082,"thread_start_us":141,"threads_started":2,"update_count":2461} I0108 16:34:04.607051 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41955619 bytes of WAL I0108 16:34:04.607125 24439 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:34:04.607172 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000001 (ops 1-12454) I0108 16:34:04.607257 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000002 (ops 12455-24841) I0108 16:34:04.607336 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000003 (ops 24842-37238) I0108 16:34:04.607434 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000004 (ops 37239-49674) I0108 16:34:04.607511 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000005 (ops 49675-62006) I0108 16:34:04.650995 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.044s user 0.004s sys 0.010s Metrics: {} I0108 16:34:04.651293 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034606 I0108 16:34:08.390995 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 3.877s user 0.879s sys 0.075s Metrics: {"bytes_written":10304612,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":947,"dirs.run_cpu_time_us":117,"dirs.run_wall_time_us":42610,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":109979,"lbm_writes_1-10_ms":25,"lbm_writes_lt_1ms":1027,"reinsert_count":0,"rows_written":585002,"thread_start_us":31,"threads_started":1,"update_count":3474} I0108 16:34:08.391165 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41933804 bytes of WAL I0108 16:34:08.391412 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:34:08.391429 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000001 (ops 1-12464) I0108 16:34:08.391490 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000002 (ops 12465-24842) I0108 16:34:08.391518 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000003 (ops 24843-37254) I0108 16:34:08.391537 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000004 (ops 37255-49701) I0108 16:34:08.391554 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000005 (ops 49702-62021) I0108 16:34:08.422865 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.031s user 0.000s sys 0.013s Metrics: {} I0108 16:34:08.423048 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 11611395 bytes on disk I0108 16:34:08.423175 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 16:34:08.423354 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.035654 I0108 16:34:08.751581 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 4.100s user 0.877s sys 0.064s Metrics: {"bytes_written":10309877,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":23475,"dirs.run_cpu_time_us":123,"dirs.run_wall_time_us":13407,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":130279,"lbm_writes_1-10_ms":44,"lbm_writes_lt_1ms":1007,"reinsert_count":0,"rows_written":584946,"thread_start_us":31,"threads_started":1,"update_count":4053} I0108 16:34:08.752462 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41949916 bytes of WAL I0108 16:34:08.752539 24439 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:34:08.752588 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000001 (ops 1-12522) I0108 16:34:08.752657 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000002 (ops 12523-25007) I0108 16:34:08.752717 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000003 (ops 25008-37523) I0108 16:34:08.752780 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000004 (ops 37524-49969) I0108 16:34:08.752841 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000005 (ops 49970-62490) I0108 16:34:08.807286 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.055s user 0.000s sys 0.013s Metrics: {} I0108 16:34:08.807448 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.035760 I0108 16:34:08.916983 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 4.597s user 0.889s sys 0.082s Metrics: {"bytes_written":10263704,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":10214,"dirs.run_cpu_time_us":111,"dirs.run_wall_time_us":64442,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":118183,"lbm_writes_1-10_ms":36,"lbm_writes_lt_1ms":1013,"reinsert_count":0,"rows_written":583303,"thread_start_us":31,"threads_started":1,"update_count":2410} I0108 16:34:08.919222 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41930421 bytes of WAL I0108 16:34:08.919294 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:34:08.919617 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000001 (ops 1-12453) I0108 16:34:08.919704 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000002 (ops 12454-24811) I0108 16:34:08.921063 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000003 (ops 24812-37224) I0108 16:34:08.921180 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000004 (ops 37225-49633) I0108 16:34:08.921545 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000005 (ops 49634-61976) I0108 16:34:09.069180 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.150s user 0.002s sys 0.011s Metrics: {} I0108 16:34:09.069324 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.035832 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 80000 Mean: 1602.2 Percentiles: 0% (min) = 728 25% = 1253 50% (med) = 1398 75% = 1580 95% = 2022 99% = 6016 99.9% = 35392 99.99% = 60896 100% (max) = 87675 I0108 16:34:23.362362 24342 linked_list-test-util.h:519] Written 4000000 rows in chain W0108 16:34:29.700570 24565 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.343s user 0.000s sys 0.000s I0108 16:34:29.701524 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 21.278s user 3.971s sys 0.302s Metrics: {"bytes_written":42278143,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":22049,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":73176,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":595194,"lbm_writes_1-10_ms":148,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":4128,"reinsert_count":0,"rows_written":2404754,"thread_start_us":77,"threads_started":2,"update_count":9973} I0108 16:34:29.702013 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 47889878 bytes on disk I0108 16:34:29.702121 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0108 16:34:29.952493 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100926640 bytes of WAL I0108 16:34:29.952615 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:34:29.952627 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000001 (ops 1-6044) I0108 16:34:29.952675 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000002 (ops 6045-11946) I0108 16:34:29.952702 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000003 (ops 11947-17819) I0108 16:34:29.952724 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000004 (ops 17820-23747) I0108 16:34:29.952746 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000005 (ops 23748-29695) I0108 16:34:29.952766 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000006 (ops 29696-35641) I0108 16:34:29.952785 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000007 (ops 35642-41444) I0108 16:34:29.952806 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000008 (ops 41445-47428) I0108 16:34:29.952828 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000009 (ops 47429-53241) I0108 16:34:29.952883 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000010 (ops 53242-59217) I0108 16:34:29.952905 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000011 (ops 59218-65056) I0108 16:34:29.952925 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000012 (ops 65057-70958) I0108 16:34:29.977190 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 21.170s user 3.933s sys 0.341s Metrics: {"bytes_written":42476538,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":22459,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":67068,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":640727,"lbm_writes_1-10_ms":160,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":4124,"reinsert_count":0,"rows_written":2411937,"thread_start_us":61,"threads_started":2,"update_count":17156} I0108 16:34:29.977574 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100892108 bytes of WAL I0108 16:34:29.977663 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:34:29.977674 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000001 (ops 1-6044) I0108 16:34:29.977722 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000002 (ops 6045-11952) I0108 16:34:29.977749 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000003 (ops 11953-17846) I0108 16:34:29.977772 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000004 (ops 17847-23772) I0108 16:34:29.977792 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000005 (ops 23773-29656) I0108 16:34:29.977813 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000006 (ops 29657-35609) I0108 16:34:29.977835 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000007 (ops 35610-41412) I0108 16:34:29.977856 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000008 (ops 41413-47383) I0108 16:34:29.977877 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000009 (ops 47384-53267) I0108 16:34:29.977922 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000010 (ops 53268-59246) I0108 16:34:29.977944 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000011 (ops 59247-65084) I0108 16:34:29.977963 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000012 (ops 65085-70967) I0108 16:34:30.022882 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.070s user 0.000s sys 0.027s Metrics: {} I0108 16:34:30.075824 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.098s user 0.001s sys 0.025s Metrics: {} I0108 16:34:30.075968 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 47974255 bytes on disk I0108 16:34:30.076797 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":701,"lbm_reads_lt_1ms":8} I0108 16:34:30.079706 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 11619297 bytes on disk I0108 16:34:30.079834 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:34:30.081231 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 11253288 bytes on disk I0108 16:34:30.081393 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0108 16:34:30.088313 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 21.019s user 3.829s sys 0.343s Metrics: {"bytes_written":42403683,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":22077,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":83940,"drs_written":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8,"lbm_write_time_us":561047,"lbm_writes_1-10_ms":151,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":4143,"reinsert_count":0,"rows_written":2415795,"spinlock_wait_cycles":29440,"thread_start_us":1427,"threads_started":2,"update_count":3858} I0108 16:34:30.088495 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100809531 bytes of WAL I0108 16:34:30.088692 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:34:30.088747 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000001 (ops 1-6069) I0108 16:34:30.088840 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000002 (ops 6070-12003) I0108 16:34:30.088909 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000003 (ops 12004-17846) I0108 16:34:30.088979 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000004 (ops 17847-23803) I0108 16:34:30.089046 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000005 (ops 23804-29786) I0108 16:34:30.089113 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000006 (ops 29787-35647) I0108 16:34:30.089179 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000007 (ops 35648-41649) I0108 16:34:30.089246 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000008 (ops 41650-47516) I0108 16:34:30.089344 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000009 (ops 47517-53508) I0108 16:34:30.089462 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000010 (ops 53509-59375) I0108 16:34:30.089601 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000011 (ops 59376-65378) I0108 16:34:30.089665 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000012 (ops 65379-71294) I0108 16:34:30.136011 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.047s user 0.002s sys 0.025s Metrics: {} I0108 16:34:30.136154 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 48171672 bytes on disk I0108 16:34:30.136739 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":456,"lbm_reads_lt_1ms":8} I0108 16:34:30.137228 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 11600423 bytes on disk I0108 16:34:30.137356 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 90000 Mean: 1615.92 Percentiles: 0% (min) = 728 25% = 1260 50% (med) = 1408 75% = 1593 95% = 2045 99% = 6168 99.9% = 35648 99.99% = 62368 100% (max) = 87675 I0108 16:34:42.371716 24342 linked_list-test-util.h:519] Written 4500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 100000 Mean: 1624.98 Percentiles: 0% (min) = 728 25% = 1268 50% (med) = 1418 75% = 1606 95% = 2058 99% = 6136 99.9% = 35520 99.99% = 62240 100% (max) = 87675 I0108 16:35:01.299621 24342 linked_list-test-util.h:519] Written 5000000 rows in chain I0108 16:35:20.481546 24342 linked_list-test-util.h:519] Written 5500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 110000 Mean: 1634.62 Percentiles: 0% (min) = 728 25% = 1276 50% (med) = 1428 75% = 1619 95% = 2070 99% = 6136 99.9% = 35520 99.99% = 60896 100% (max) = 87675 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 120000 Mean: 1642.95 Percentiles: 0% (min) = 728 25% = 1283 50% (med) = 1436 75% = 1628 95% = 2084 99% = 6172 99.9% = 35520 99.99% = 59552 100% (max) = 87675 I0108 16:35:39.796090 24342 linked_list-test-util.h:519] Written 6000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 130000 Mean: 1651.51 Percentiles: 0% (min) = 728 25% = 1290 50% (med) = 1445 75% = 1638 95% = 2096 99% = 6204 99.9% = 35520 99.99% = 59488 100% (max) = 87675 I0108 16:35:59.221300 24342 linked_list-test-util.h:519] Written 6500000 rows in chain I0108 16:36:04.473870 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033395 I0108 16:36:04.610844 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033391 I0108 16:36:04.664036 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033349 I0108 16:36:07.972306 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 3.498s user 0.827s sys 0.054s Metrics: {"bytes_written":9561102,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":21469,"dirs.run_cpu_time_us":94,"dirs.run_wall_time_us":152571,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":106027,"lbm_writes_1-10_ms":23,"lbm_writes_lt_1ms":949,"reinsert_count":0,"rows_written":539569,"thread_start_us":30,"threads_started":1,"update_count":1298} I0108 16:36:07.973438 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41964663 bytes of WAL I0108 16:36:07.973608 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:36:07.973723 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000006 (ops 62487-74969) I0108 16:36:07.974484 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000007 (ops 74970-87505) I0108 16:36:07.974553 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000008 (ops 87506-99975) I0108 16:36:07.975175 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000009 (ops 99976-112449) I0108 16:36:07.975253 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000010 (ops 112450-124873) I0108 16:36:08.016240 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.043s user 0.002s sys 0.011s Metrics: {} I0108 16:36:08.017122 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 10738711 bytes on disk I0108 16:36:08.017402 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 16:36:08.017499 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.003865 I0108 16:36:08.049312 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 3.438s user 0.829s sys 0.054s Metrics: {"bytes_written":9605308,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9016,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":23163,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":93692,"lbm_writes_1-10_ms":26,"lbm_writes_lt_1ms":950,"reinsert_count":0,"rows_written":541847,"thread_start_us":30,"threads_started":1,"update_count":1804} I0108 16:36:08.049937 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41994890 bytes of WAL I0108 16:36:08.050112 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:36:08.050156 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000006 (ops 62301-74778) I0108 16:36:08.050241 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000007 (ops 74779-87194) I0108 16:36:08.050299 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000008 (ops 87195-99653) I0108 16:36:08.050361 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000009 (ops 99654-112094) I0108 16:36:08.050449 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000010 (ops 112095-124571) I0108 16:36:08.092308 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.042s user 0.004s sys 0.009s Metrics: {} I0108 16:36:08.093863 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 10777666 bytes on disk I0108 16:36:08.093988 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:36:08.094075 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.003848 I0108 16:36:08.663815 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 4.000s user 0.823s sys 0.054s Metrics: {"bytes_written":9574992,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":5524,"dirs.run_cpu_time_us":72,"dirs.run_wall_time_us":36581,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":90029,"lbm_writes_1-10_ms":23,"lbm_writes_lt_1ms":950,"reinsert_count":0,"rows_written":539817,"thread_start_us":31,"threads_started":1,"update_count":2173} I0108 16:36:08.663938 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41963060 bytes of WAL I0108 16:36:08.663980 24439 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:36:08.663987 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000006 (ops 62007-74425) I0108 16:36:08.664026 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000007 (ops 74426-86807) I0108 16:36:08.664053 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000008 (ops 86808-99177) I0108 16:36:08.664072 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000009 (ops 99178-111591) I0108 16:36:08.664088 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000010 (ops 111592-124003) I0108 16:36:08.770519 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.107s user 0.001s sys 0.013s Metrics: {} I0108 16:36:08.772296 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 10736834 bytes on disk I0108 16:36:08.772701 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:36:08.772795 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.003855 I0108 16:36:15.111421 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 7.017s user 1.487s sys 0.153s Metrics: {"bytes_written":19409673,"cfile_cache_hit":52,"cfile_cache_hit_bytes":6151313,"cfile_cache_miss":2574,"cfile_cache_miss_bytes":71653335,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":9730,"dirs.run_cpu_time_us":93,"dirs.run_wall_time_us":24866,"drs_written":1,"lbm_read_time_us":182492,"lbm_reads_1-10_ms":43,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":2565,"lbm_write_time_us":286548,"lbm_writes_1-10_ms":48,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":1913,"num_input_rowsets":2,"rows_written":1107417,"spinlock_wait_cycles":4352,"thread_start_us":270,"threads_started":9} I0108 16:36:15.111652 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.035200 I0108 16:36:15.390257 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 7.373s user 1.493s sys 0.142s Metrics: {"bytes_written":19383032,"cfile_cache_hit":51,"cfile_cache_hit_bytes":6000910,"cfile_cache_miss":2571,"cfile_cache_miss_bytes":71678548,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":5435,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":145459,"drs_written":1,"lbm_read_time_us":261404,"lbm_reads_1-10_ms":37,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":2568,"lbm_write_time_us":344762,"lbm_writes_1-10_ms":58,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":1900,"num_input_rowsets":2,"rows_written":1106024,"spinlock_wait_cycles":25742080,"thread_start_us":228,"threads_started":9} I0108 16:36:15.390363 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 22491402 bytes on disk I0108 16:36:15.390455 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:36:15.390525 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.035132 I0108 16:36:15.833645 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 7.061s user 1.506s sys 0.148s Metrics: {"bytes_written":19400976,"cfile_cache_hit":53,"cfile_cache_hit_bytes":6157276,"cfile_cache_miss":2571,"cfile_cache_miss_bytes":71593232,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":495,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":40365,"drs_written":1,"lbm_read_time_us":162110,"lbm_reads_1-10_ms":31,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":2574,"lbm_write_time_us":220346,"lbm_writes_1-10_ms":51,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":1913,"num_input_rowsets":2,"rows_written":1106899,"spinlock_wait_cycles":15435008,"thread_start_us":1743,"threads_started":9} I0108 16:36:15.833812 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.035301 I0108 16:36:18.340329 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 2.950s user 0.851s sys 0.064s Metrics: {"bytes_written":10034591,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":5182,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":46428,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":136711,"lbm_writes_1-10_ms":28,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":993,"reinsert_count":0,"rows_written":567640,"thread_start_us":31,"threads_started":1,"update_count":22} I0108 16:36:18.340569 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 42003694 bytes of WAL I0108 16:36:18.340638 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:36:18.340680 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000006 (ops 61977-74411) I0108 16:36:18.340802 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000007 (ops 74412-86768) I0108 16:36:18.341280 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000008 (ops 86769-99145) I0108 16:36:18.341622 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000009 (ops 99146-111562) I0108 16:36:18.342118 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000010 (ops 111563-123976) I0108 16:36:18.409868 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.069s user 0.000s sys 0.015s Metrics: {} I0108 16:36:18.410030 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.003406 I0108 16:36:18.426734 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 3.315s user 0.863s sys 0.065s Metrics: {"bytes_written":10018713,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":19561,"dirs.run_cpu_time_us":63,"dirs.run_wall_time_us":63,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":253667,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":7,"lbm_writes_lt_1ms":988,"reinsert_count":0,"rows_written":564400,"thread_start_us":30,"threads_started":1,"update_count":3414} I0108 16:36:18.431671 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 22535152 bytes on disk I0108 16:36:18.442493 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.010s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10385,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":2} I0108 16:36:18.442592 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 11233685 bytes on disk I0108 16:36:18.442972 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":243,"lbm_reads_lt_1ms":4} I0108 16:36:18.443073 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.003420 I0108 16:36:19.152699 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 3.319s user 0.858s sys 0.072s Metrics: {"bytes_written":10050118,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":146,"dirs.run_cpu_time_us":52,"dirs.run_wall_time_us":86,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":203942,"lbm_writes_1-10_ms":33,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":984,"reinsert_count":0,"rows_written":567517,"spinlock_wait_cycles":768,"thread_start_us":54,"threads_started":1,"update_count":1542} I0108 16:36:19.152882 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41992733 bytes of WAL I0108 16:36:19.153329 24439 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:36:19.153381 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000006 (ops 62491-74985) I0108 16:36:19.153481 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000007 (ops 74986-87468) I0108 16:36:19.153587 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000008 (ops 87469-99909) I0108 16:36:19.153658 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000009 (ops 99910-112355) I0108 16:36:19.153719 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000010 (ops 112356-124825) I0108 16:36:19.201346 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.048s user 0.000s sys 0.011s Metrics: {} I0108 16:36:19.202682 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 22509927 bytes on disk I0108 16:36:19.214792 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11977,"lbm_reads_1-10_ms":2,"lbm_reads_lt_1ms":2} I0108 16:36:19.214920 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.003389 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 140000 Mean: 1667.74 Percentiles: 0% (min) = 728 25% = 1296 50% (med) = 1454 75% = 1651 95% = 2126 99% = 6312 99.9% = 36064 99.99% = 60896 100% (max) = 87675 I0108 16:36:19.667351 24342 linked_list-test-util.h:519] Written 7000000 rows in chain I0108 16:36:25.387609 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 6.977s user 1.550s sys 0.169s Metrics: {"bytes_written":20163715,"cfile_cache_hit":53,"cfile_cache_hit_bytes":6472242,"cfile_cache_miss":2673,"cfile_cache_miss_bytes":74413698,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":22665,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":15499,"drs_written":1,"lbm_read_time_us":657233,"lbm_reads_1-10_ms":54,"lbm_reads_10-100_ms":8,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":2649,"lbm_write_time_us":262384,"lbm_writes_1-10_ms":53,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":1989,"num_input_rowsets":2,"rows_written":1150943,"spinlock_wait_cycles":32375424,"thread_start_us":395,"threads_started":9} I0108 16:36:25.387797 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 23465504 bytes on disk I0108 16:36:25.387925 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:36:26.070569 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 6.856s user 1.510s sys 0.171s Metrics: {"bytes_written":20185909,"cfile_cache_hit":51,"cfile_cache_hit_bytes":6171308,"cfile_cache_miss":2675,"cfile_cache_miss_bytes":74839084,"cfile_init":10,"delta_iterators_relevant":8,"dirs.queue_time_us":3713,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":74228,"drs_written":1,"lbm_read_time_us":647034,"lbm_reads_1-10_ms":42,"lbm_reads_10-100_ms":16,"lbm_reads_lt_1ms":2657,"lbm_write_time_us":241468,"lbm_writes_1-10_ms":77,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":1969,"num_input_rowsets":2,"rows_written":1152463,"spinlock_wait_cycles":46535552,"thread_start_us":1425,"threads_started":9} I0108 16:36:26.070735 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 23499578 bytes on disk I0108 16:36:26.070967 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:36:26.511261 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 8.067s user 1.564s sys 0.174s Metrics: {"bytes_written":20137238,"cfile_cache_hit":50,"cfile_cache_hit_bytes":5814435,"cfile_cache_miss":2670,"cfile_cache_miss_bytes":74966421,"cfile_init":9,"delta_iterators_relevant":8,"dirs.queue_time_us":6760,"dirs.run_cpu_time_us":115,"dirs.run_wall_time_us":51493,"drs_written":1,"lbm_read_time_us":992648,"lbm_reads_1-10_ms":46,"lbm_reads_10-100_ms":14,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":2643,"lbm_write_time_us":244953,"lbm_writes_1-10_ms":70,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":1972,"num_input_rowsets":2,"rows_written":1149402,"spinlock_wait_cycles":5049984,"thread_start_us":625,"threads_started":9} I0108 16:36:26.511445 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41939473 bytes of WAL I0108 16:36:26.511559 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:36:26.511602 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000006 (ops 62022-74451) I0108 16:36:26.511726 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000007 (ops 74452-86835) I0108 16:36:26.511816 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000008 (ops 86836-99238) I0108 16:36:26.511900 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000009 (ops 99239-111591) I0108 16:36:26.511965 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000010 (ops 111592-124016) I0108 16:36:26.537521 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.026s user 0.000s sys 0.014s Metrics: {} I0108 16:36:26.537606 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 23419487 bytes on disk I0108 16:36:26.569849 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.032s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":32140,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0108 16:36:29.822213 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033367 I0108 16:36:30.074784 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033360 I0108 16:36:30.141418 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033348 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 150000 Mean: 1681.2 Percentiles: 0% (min) = 728 25% = 1301 50% (med) = 1460 75% = 1658 95% = 2146 99% = 6456 99.9% = 37152 99.99% = 63264 100% (max) = 258471 I0108 16:36:40.045864 24342 linked_list-test-util.h:519] Written 7500000 rows in chain I0108 16:36:49.530864 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 19.456s user 3.986s sys 0.283s Metrics: {"bytes_written":42921428,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":24549,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":72422,"drs_written":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8,"lbm_write_time_us":593415,"lbm_writes_1-10_ms":130,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":4181,"reinsert_count":0,"rows_written":2425578,"spinlock_wait_cycles":28928,"thread_start_us":64,"threads_started":2,"update_count":7060} I0108 16:36:49.530979 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=1.004941 I0108 16:36:50.763447 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 20.622s user 4.030s sys 0.286s Metrics: {"bytes_written":42899199,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":11749,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":74753,"drs_written":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8,"lbm_write_time_us":600591,"lbm_writes_1-10_ms":146,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":4162,"reinsert_count":0,"rows_written":2423265,"spinlock_wait_cycles":31488,"thread_start_us":70,"threads_started":2,"update_count":8605} I0108 16:36:50.764091 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100791721 bytes of WAL I0108 16:36:50.764295 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:36:50.766520 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000013 (ops 71295-77294) I0108 16:36:50.766954 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000014 (ops 77295-83233) I0108 16:36:50.767019 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000015 (ops 83234-89078) I0108 16:36:50.767077 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000016 (ops 89079-95031) I0108 16:36:50.767175 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000017 (ops 95032-100923) I0108 16:36:50.767232 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000018 (ops 100924-106895) I0108 16:36:50.767285 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000019 (ops 106896-112850) I0108 16:36:50.767338 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000020 (ops 112851-118719) I0108 16:36:50.767388 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000021 (ops 118720-124701) I0108 16:36:50.767496 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000022 (ops 124702-130667) I0108 16:36:50.767554 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000023 (ops 130668-136488) I0108 16:36:50.767602 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000024 (ops 136489-142490) I0108 16:36:50.839534 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.075s user 0.003s sys 0.027s Metrics: {} I0108 16:36:50.846093 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=1.004926 I0108 16:36:51.093575 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 21.271s user 4.020s sys 0.298s Metrics: {"bytes_written":42888218,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":3194,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":83685,"drs_written":2,"lbm_read_time_us":27,"lbm_reads_lt_1ms":8,"lbm_write_time_us":953271,"lbm_writes_1-10_ms":170,"lbm_writes_10-100_ms":5,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":4141,"reinsert_count":0,"rows_written":2427365,"spinlock_wait_cycles":29312,"thread_start_us":69,"threads_started":2,"update_count":1664} I0108 16:36:51.098137 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100658069 bytes of WAL I0108 16:36:51.098232 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:36:51.098273 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000013 (ops 70959-76892) I0108 16:36:51.098353 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000014 (ops 76893-82792) I0108 16:36:51.098419 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000015 (ops 82793-88621) I0108 16:36:51.098510 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000016 (ops 88622-94530) I0108 16:36:51.098580 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000017 (ops 94531-100444) I0108 16:36:51.098639 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000018 (ops 100445-106388) I0108 16:36:51.098700 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000019 (ops 106389-112307) I0108 16:36:51.098755 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000020 (ops 112308-118172) I0108 16:36:51.098814 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000021 (ops 118173-124126) I0108 16:36:51.098906 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000022 (ops 124127-130053) I0108 16:36:51.098963 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000023 (ops 130054-135856) I0108 16:36:51.099015 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000024 (ops 135857-141751) I0108 16:36:51.350870 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.253s user 0.000s sys 0.039s Metrics: {} I0108 16:36:51.352246 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=1.004997 I0108 16:37:00.349815 24342 linked_list-test-util.h:528] Sample at HT timestamp: 6465659782548680704 Inserted count: 7989850 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 160000 Mean: 1696.21 Percentiles: 0% (min) = 728 25% = 1308 50% (med) = 1467 75% = 1666 95% = 2170 99% = 6584 99.9% = 37984 99.99% = 69568 100% (max) = 258471 I0108 16:37:00.749554 24342 linked_list-test-util.h:519] Written 8000000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 170000 Mean: 1708.24 Percentiles: 0% (min) = 728 25% = 1313 50% (med) = 1473 75% = 1675 95% = 2194 99% = 6644 99.9% = 38944 99.99% = 72960 100% (max) = 258471 I0108 16:37:21.354174 24342 linked_list-test-util.h:519] Written 8500000 rows in chain I0108 16:37:24.363766 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 34.833s user 6.496s sys 0.713s Metrics: {"bytes_written":84679542,"cfile_cache_hit":260,"cfile_cache_hit_bytes":36741168,"cfile_cache_miss":11152,"cfile_cache_miss_bytes":303608666,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":22019,"dirs.run_cpu_time_us":270,"dirs.run_wall_time_us":82727,"drs_written":3,"lbm_read_time_us":2030781,"lbm_reads_1-10_ms":270,"lbm_reads_10-100_ms":29,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":10935,"lbm_write_time_us":1164004,"lbm_writes_1-10_ms":303,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":8262,"num_input_rowsets":4,"rows_written":4837515,"spinlock_wait_cycles":46825600,"thread_start_us":1107,"threads_started":11} I0108 16:37:24.364187 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100676675 bytes of WAL I0108 16:37:24.368345 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:37:24.368366 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000013 (ops 70968-76894) I0108 16:37:24.368429 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000014 (ops 76895-82792) I0108 16:37:24.368465 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000015 (ops 82793-88652) I0108 16:37:24.368496 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000016 (ops 88653-94555) I0108 16:37:24.368530 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000017 (ops 94556-100442) I0108 16:37:24.368553 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000018 (ops 100443-106387) I0108 16:37:24.368578 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000019 (ops 106388-112309) I0108 16:37:24.368598 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000020 (ops 112310-118171) I0108 16:37:24.368619 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000021 (ops 118172-124059) I0108 16:37:24.368639 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000022 (ops 124060-129990) I0108 16:37:24.368659 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000023 (ops 129991-135855) I0108 16:37:24.368674 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000024 (ops 135856-141813) I0108 16:37:24.429432 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.061s user 0.003s sys 0.025s Metrics: {} I0108 16:37:24.429651 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 98972095 bytes on disk I0108 16:37:24.563591 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.134s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":133709,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":6} I0108 16:37:24.818334 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 33.466s user 6.516s sys 0.684s Metrics: {"bytes_written":84586312,"cfile_cache_hit":256,"cfile_cache_hit_bytes":36035810,"cfile_cache_miss":11148,"cfile_cache_miss_bytes":303876002,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":8206,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":92079,"drs_written":3,"lbm_read_time_us":2536159,"lbm_reads_1-10_ms":271,"lbm_reads_10-100_ms":38,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":10921,"lbm_write_time_us":1048862,"lbm_writes_1-10_ms":274,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":8286,"num_input_rowsets":4,"rows_written":4832119,"spinlock_wait_cycles":177876480,"thread_start_us":93,"threads_started":3} I0108 16:37:24.818467 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 98850092 bytes on disk I0108 16:37:24.818608 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":23,"lbm_reads_lt_1ms":12} I0108 16:37:24.984833 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 34.139s user 6.575s sys 0.670s Metrics: {"bytes_written":84705377,"cfile_cache_hit":255,"cfile_cache_hit_bytes":35986237,"cfile_cache_miss":11157,"cfile_cache_miss_bytes":304429131,"cfile_init":21,"delta_iterators_relevant":16,"dirs.queue_time_us":17153,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":107635,"drs_written":3,"lbm_read_time_us":2546414,"lbm_reads_1-10_ms":240,"lbm_reads_10-100_ms":26,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":10969,"lbm_write_time_us":1321679,"lbm_writes_1-10_ms":302,"lbm_writes_10-100_ms":17,"lbm_writes_lt_1ms":8258,"num_input_rowsets":4,"rows_written":4839060,"spinlock_wait_cycles":254933120,"thread_start_us":483,"threads_started":11} I0108 16:37:24.985050 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 99004383 bytes on disk I0108 16:37:25.054308 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.069s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":69043,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":8} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 180000 Mean: 1722.03 Percentiles: 0% (min) = 728 I0108 16:37:42.776893 24342 linked_list-test-util.h:519] Written 9000000 rows in chain 25% = 1315 50% (med) = 1475 75% = 1676 95% = 2196 99% = 6772 99.9% = 40320 99.99% = 79872 100% (max) = 398140 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 190000 Mean: 1725.54 Percentiles: 0% (min) = 728 25% = 1318 50% (med) = 1478 75% = 1679 95% = 2198 99% = 6788 99.9% = 40672 99.99% = 79424 100% (max) = 398140 I0108 16:38:02.407656 24342 linked_list-test-util.h:519] Written 9500000 rows in chain I0108 16:38:08.094918 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033367 I0108 16:38:08.103818 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033348 I0108 16:38:08.863179 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033389 I0108 16:38:12.210250 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 4.106s user 0.751s sys 0.064s Metrics: {"bytes_written":8846443,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":14131,"dirs.run_cpu_time_us":110,"dirs.run_wall_time_us":25530,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":119783,"lbm_writes_1-10_ms":37,"lbm_writes_lt_1ms":869,"reinsert_count":0,"rows_written":501076,"thread_start_us":63,"threads_started":1,"update_count":3434} I0108 16:38:12.210537 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41932711 bytes of WAL I0108 16:38:12.210613 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:38:12.210661 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000011 (ops 124572-137020) I0108 16:38:12.210747 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000012 (ops 137021-149414) I0108 16:38:12.210808 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000013 (ops 149415-161859) I0108 16:38:12.211246 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000014 (ops 161860-174310) I0108 16:38:12.211309 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000015 (ops 174311-186654) I0108 16:38:12.255735 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.045s user 0.002s sys 0.013s Metrics: {} I0108 16:38:12.255885 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 9920610 bytes on disk I0108 16:38:12.256016 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:38:12.256104 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.000965 I0108 16:38:12.406285 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 4.311s user 0.745s sys 0.059s Metrics: {"bytes_written":8838735,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":9560,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":30996,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":226269,"lbm_writes_1-10_ms":47,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":853,"reinsert_count":0,"rows_written":500908,"thread_start_us":36,"threads_started":1,"update_count":3349} I0108 16:38:12.407124 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.000967 I0108 16:38:13.033967 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 4.170s user 0.784s sys 0.061s Metrics: {"bytes_written":8871927,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":82,"dirs.run_cpu_time_us":130,"dirs.run_wall_time_us":65617,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":149889,"lbm_writes_1-10_ms":35,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":873,"reinsert_count":0,"rows_written":503247,"thread_start_us":38,"threads_started":1,"update_count":2382} I0108 16:38:13.034155 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 42000894 bytes of WAL I0108 16:38:13.035403 24439 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:38:13.035601 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000011 (ops 124004-136447) I0108 16:38:13.035679 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000012 (ops 136448-148873) I0108 16:38:13.035836 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000013 (ops 148874-161244) I0108 16:38:13.035903 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000014 (ops 161245-173620) I0108 16:38:13.038120 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000015 (ops 173621-186054) I0108 16:38:13.156936 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.121s user 0.000s sys 0.022s Metrics: {} I0108 16:38:13.158080 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.000944 I0108 16:38:22.113323 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 9.706s user 2.192s sys 0.215s Metrics: {"bytes_written":28051547,"cfile_cache_hit":75,"cfile_cache_hit_bytes":9928613,"cfile_cache_miss":3717,"cfile_cache_miss_bytes":103597907,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":52448,"drs_written":1,"lbm_read_time_us":496719,"lbm_reads_1-10_ms":81,"lbm_reads_10-100_ms":10,"lbm_reads_lt_1ms":3662,"lbm_write_time_us":293487,"lbm_writes_1-10_ms":61,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":2785,"num_input_rowsets":2,"rows_written":1606932,"spinlock_wait_cycles":8225152,"thread_start_us":267,"threads_started":9} I0108 16:38:22.113667 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41962526 bytes of WAL I0108 16:38:22.113739 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:38:22.113781 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000011 (ops 124874-137381) I0108 16:38:22.114468 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000012 (ops 137382-149824) I0108 16:38:22.114537 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000013 (ops 149825-162339) I0108 16:38:22.114598 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000014 (ops 162340-174737) I0108 16:38:22.115170 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000015 (ops 174738-187236) I0108 16:38:22.152892 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.039s user 0.000s sys 0.014s Metrics: {} I0108 16:38:22.153062 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 33291173 bytes on disk I0108 16:38:22.187899 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.035s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":34698,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} I0108 16:38:22.188031 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034402 I0108 16:38:22.672348 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 10.416s user 2.141s sys 0.262s Metrics: {"bytes_written":28080480,"cfile_cache_hit":80,"cfile_cache_hit_bytes":10720300,"cfile_cache_miss":3716,"cfile_cache_miss_bytes":102938992,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":6072,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":31884,"drs_written":1,"lbm_read_time_us":959000,"lbm_reads_1-10_ms":79,"lbm_reads_10-100_ms":14,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":3653,"lbm_write_time_us":372115,"lbm_writes_1-10_ms":88,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":2761,"num_input_rowsets":2,"rows_written":1608493,"spinlock_wait_cycles":16092544,"thread_start_us":441,"threads_started":9} I0108 16:38:22.672458 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 33333162 bytes on disk I0108 16:38:22.672574 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:38:22.672655 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.034513 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 200000 Mean: 1733.88 Percentiles: 0% (min) = 728 25% = 1321 50% (med) = 1481 75% = 1683 95% = 2212 99% = 6864 99.9% = 41632 99.99% = 80000 100% (max) = 398140 I0108 16:38:22.958241 24342 linked_list-test-util.h:519] Written 10000000 rows in chain I0108 16:38:23.750663 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 10.592s user 2.214s sys 0.216s Metrics: {"bytes_written":28107825,"cfile_cache_hit":85,"cfile_cache_hit_bytes":11419642,"cfile_cache_miss":3715,"cfile_cache_miss_bytes":102331498,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":12538,"dirs.run_cpu_time_us":95,"dirs.run_wall_time_us":81434,"drs_written":1,"lbm_read_time_us":971885,"lbm_reads_1-10_ms":68,"lbm_reads_10-100_ms":14,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3668,"lbm_write_time_us":344140,"lbm_writes_1-10_ms":74,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":2778,"num_input_rowsets":2,"rows_written":1610146,"spinlock_wait_cycles":11008,"thread_start_us":3566,"threads_started":9} I0108 16:38:23.750838 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 33364378 bytes on disk I0108 16:38:23.754547 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:38:23.754711 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034612 I0108 16:38:25.212360 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 3.024s user 0.774s sys 0.054s Metrics: {"bytes_written":9057321,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":12871,"dirs.run_cpu_time_us":85,"dirs.run_wall_time_us":34340,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":146931,"lbm_writes_1-10_ms":28,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":897,"reinsert_count":0,"rows_written":514261,"thread_start_us":30,"threads_started":1,"update_count":2007} I0108 16:38:25.217665 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 42030608 bytes of WAL I0108 16:38:25.217986 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:38:25.218040 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000011 (ops 123977-136447) I0108 16:38:25.218140 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000012 (ops 136448-148824) I0108 16:38:25.218221 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000013 (ops 148825-161259) I0108 16:38:25.218288 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000014 (ops 161260-173602) I0108 16:38:25.218783 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000015 (ops 173603-186025) I0108 16:38:25.241221 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.023s user 0.000s sys 0.013s Metrics: {} I0108 16:38:25.249650 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.000834 I0108 16:38:26.476074 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 3.803s user 0.781s sys 0.076s Metrics: {"bytes_written":9062410,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":25923,"dirs.run_cpu_time_us":50,"dirs.run_wall_time_us":50,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":229828,"lbm_writes_1-10_ms":34,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":893,"reinsert_count":0,"rows_written":515891,"thread_start_us":30,"threads_started":1,"update_count":24} I0108 16:38:26.476285 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 42016020 bytes of WAL I0108 16:38:26.476361 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:38:26.476864 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000011 (ops 124017-136492) I0108 16:38:26.476953 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000012 (ops 136493-148887) I0108 16:38:26.477028 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000013 (ops 148888-161323) I0108 16:38:26.477100 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000014 (ops 161324-173688) I0108 16:38:26.477205 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000015 (ops 173689-186128) I0108 16:38:26.508571 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.032s user 0.000s sys 0.015s Metrics: {} I0108 16:38:26.508724 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.000816 I0108 16:38:27.033937 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 3.275s user 0.793s sys 0.056s Metrics: {"bytes_written":9117191,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":58101,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":147996,"lbm_writes_1-10_ms":27,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":905,"reinsert_count":0,"rows_written":518760,"thread_start_us":35,"threads_started":1,"update_count":297} I0108 16:38:27.034051 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41991527 bytes of WAL I0108 16:38:27.034092 24439 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:38:27.034104 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000011 (ops 124826-137343) I0108 16:38:27.034134 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000012 (ops 137344-149758) I0108 16:38:27.034157 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000013 (ops 149759-162290) I0108 16:38:27.034174 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000014 (ops 162291-174669) I0108 16:38:27.034198 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000015 (ops 174670-187143) I0108 16:38:27.080703 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.047s user 0.000s sys 0.014s Metrics: {} I0108 16:38:27.080850 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.000788 I0108 16:38:29.770797 26317 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=601060, attempt_no=0}) took 1029ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":12,"cfile_cache_hit":2820,"cfile_cache_hit_bytes":11605608,"num_ops":17179,"prepare.queue_time_us":166,"prepare.run_cpu_time_us":18035,"prepare.run_wall_time_us":62368,"raft.queue_time_us":44,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":180,"replication_time_us":10883,"spinlock_wait_cycles":3918080}]]} I0108 16:38:36.816023 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 11.566s user 2.257s sys 0.248s Metrics: {"bytes_written":29059393,"cfile_cache_hit":81,"cfile_cache_hit_bytes":10858684,"cfile_cache_miss":3851,"cfile_cache_miss_bytes":106878166,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":7609,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":37017,"drs_written":1,"lbm_read_time_us":808600,"lbm_reads_1-10_ms":108,"lbm_reads_10-100_ms":16,"lbm_reads_lt_1ms":3763,"lbm_write_time_us":409797,"lbm_writes_1-10_ms":86,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":2859,"num_input_rowsets":2,"rows_written":1665204,"spinlock_wait_cycles":128273152,"thread_start_us":1025,"threads_started":9} I0108 16:38:36.816128 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 34571398 bytes on disk I0108 16:38:36.821604 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4} I0108 16:38:38.278017 26394 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.131:58284 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=613176, attempt_no=0}) took 1358ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":131,"cfile_cache_hit":75,"cfile_cache_hit_bytes":317381,"cfile_cache_miss":889,"cfile_cache_miss_bytes":3650234,"lbm_read_time_us":1331447,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":12,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":868,"num_ops":4104,"prepare.queue_time_us":19,"prepare.run_cpu_time_us":3213,"prepare.run_wall_time_us":3643,"raft.queue_time_us":26,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":86,"replication_time_us":1222,"spinlock_wait_cycles":70656}]]} I0108 16:38:38.534875 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 12.026s user 2.099s sys 0.272s Metrics: {"bytes_written":29059511,"cfile_cache_hit":81,"cfile_cache_hit_bytes":10842409,"cfile_cache_miss":3851,"cfile_cache_miss_bytes":106887759,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":868,"dirs.run_cpu_time_us":106,"dirs.run_wall_time_us":153025,"drs_written":1,"lbm_read_time_us":923336,"lbm_reads_1-10_ms":82,"lbm_reads_10-100_ms":12,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":3791,"lbm_write_time_us":451253,"lbm_writes_1-10_ms":128,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2819,"num_input_rowsets":2,"rows_written":1665293,"spinlock_wait_cycles":74600320,"thread_start_us":430,"threads_started":9} I0108 16:38:38.535166 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 34571607 bytes on disk I0108 16:38:38.536283 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 16:38:38.592876 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 11.512s user 2.252s sys 0.216s Metrics: {"bytes_written":29165562,"cfile_cache_hit":83,"cfile_cache_hit_bytes":11147538,"cfile_cache_miss":3861,"cfile_cache_miss_bytes":107017534,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":739,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":40595,"drs_written":1,"lbm_read_time_us":787754,"lbm_reads_1-10_ms":75,"lbm_reads_10-100_ms":15,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3806,"lbm_write_time_us":398066,"lbm_writes_1-10_ms":103,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":2858,"mutex_wait_us":1,"num_input_rowsets":2,"rows_written":1671223,"spinlock_wait_cycles":7680,"thread_start_us":698,"threads_started":9} I0108 16:38:38.592979 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 34705829 bytes on disk I0108 16:38:38.629631 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.037s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":36545,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 210000 Mean: 1745.33 Percentiles: 0% (min) = 728 25% = 1323 50% (med) = 1484 75% = 1687 95% = 2224 99% = 6920 99.9% = 41984 99.99% = 83264 100% (max) = 593952 I0108 16:38:44.267355 24342 linked_list-test-util.h:519] Written 10500000 rows in chain I0108 16:38:49.638509 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033363 I0108 16:38:50.833204 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033353 I0108 16:38:51.295944 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033390 ------------------------------------------------------------ I0108 16:39:04.435278 24342 linked_list-test-util.h:519] Written 11000000 rows in chain Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 220000 Mean: 1750.45 Percentiles: 0% (min) = 728 25% = 1326 50% (med) = 1487 75% = 1691 95% = 2234 99% = 6956 99.9% = 41984 99.99% = 80256 100% (max) = 593952 W0108 16:39:08.725572 24439 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.357s user 0.000s sys 0.000s I0108 16:39:08.726527 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 19.088s user 3.672s sys 0.277s Metrics: {"bytes_written":39737465,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":45349,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":105615,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":654193,"lbm_writes_1-10_ms":149,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":3859,"reinsert_count":0,"rows_written":2254574,"thread_start_us":59,"threads_started":2,"update_count":17436} I0108 16:39:08.726783 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 45029658 bytes on disk I0108 16:39:08.727689 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 16:39:08.727797 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=1.002760 I0108 16:39:10.970896 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 20.138s user 3.715s sys 0.302s Metrics: {"bytes_written":39918903,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":20614,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":226006,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":801377,"lbm_writes_1-10_ms":159,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3884,"reinsert_count":0,"rows_written":2273455,"thread_start_us":139,"threads_started":2,"update_count":3720} I0108 16:39:10.983955 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 101024092 bytes of WAL I0108 16:39:10.984066 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:39:10.984102 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000025 (ops 142491-148381) I0108 16:39:10.984176 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000026 (ops 148382-154343) I0108 16:39:10.984236 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000027 (ops 154344-160219) I0108 16:39:10.984292 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000028 (ops 160220-166213) I0108 16:39:10.984359 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000029 (ops 166214-172165) I0108 16:39:10.984436 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000030 (ops 172166-178020) I0108 16:39:10.984498 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000031 (ops 178021-183913) I0108 16:39:10.984555 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000032 (ops 183914-189791) I0108 16:39:10.984647 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000033 (ops 189792-195792) I0108 16:39:10.984702 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000034 (ops 195793-201726) I0108 16:39:10.984751 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000035 (ops 201727-207720) I0108 16:39:10.984800 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000036 (ops 207721-213697) I0108 16:39:11.166115 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.177s user 0.000s sys 0.033s Metrics: {} I0108 16:39:11.166682 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=1.002570 I0108 16:39:11.454020 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 20.158s user 3.713s sys 0.261s Metrics: {"bytes_written":40291605,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":22033,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":149224,"drs_written":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8,"lbm_write_time_us":666787,"lbm_writes_1-10_ms":140,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3931,"reinsert_count":0,"rows_written":2289230,"thread_start_us":58,"threads_started":2,"update_count":12554} I0108 16:39:11.454190 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=1.002410 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 230000 Mean: 1757.36 Percentiles: 0% (min) = 728 25% = 1329 50% (med) = 1490 75% = 1695 95% = 2244 99% = 7016 99.9% = 42688 99.99% = 80256 100% (max) = 593952 I0108 16:39:25.008324 24342 linked_list-test-util.h:519] Written 11500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 240000 Mean: 1763.86 Percentiles: 0% (min) = 728 25% = 1333 50% (med) = 1494 75% = 1700 95% = 2258 99% = 7080 99.9% = 43296 99.99% = 81216 100% (max) = 593952 I0108 16:39:45.723778 24342 linked_list-test-util.h:519] Written 12000000 rows in chain W0108 16:40:02.807624 24439 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.648s user 0.000s sys 0.000s I0108 16:40:02.810659 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 54.083s user 9.495s sys 1.075s Metrics: {"bytes_written":123671318,"cfile_cache_hit":371,"cfile_cache_hit_bytes":52158421,"cfile_cache_miss":16349,"cfile_cache_miss_bytes":450481789,"cfile_init":21,"delta_iterators_relevant":14,"dirs.queue_time_us":27956,"dirs.run_cpu_time_us":405,"dirs.run_wall_time_us":165753,"drs_written":4,"lbm_read_time_us":4175958,"lbm_reads_1-10_ms":477,"lbm_reads_10-100_ms":57,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":15895,"lbm_write_time_us":1779330,"lbm_writes_1-10_ms":437,"lbm_writes_10-100_ms":14,"lbm_writes_lt_1ms":12115,"num_input_rowsets":5,"rows_written":7092089,"spinlock_wait_cycles":467723904,"thread_start_us":124,"threads_started":4} I0108 16:40:02.810798 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100998050 bytes of WAL I0108 16:40:02.810961 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:40:02.810974 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000025 (ops 141814-147713) I0108 16:40:02.811010 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000026 (ops 147714-153567) I0108 16:40:02.811035 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000027 (ops 153568-159542) I0108 16:40:02.811059 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000028 (ops 159543-165463) I0108 16:40:02.811081 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000029 (ops 165464-171472) I0108 16:40:02.811100 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000030 (ops 171473-177346) I0108 16:40:02.811122 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000031 (ops 177347-183282) I0108 16:40:02.811142 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000032 (ops 183283-189218) I0108 16:40:02.811162 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000033 (ops 189219-195056) I0108 16:40:02.811182 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000034 (ops 195057-200985) I0108 16:40:02.811203 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000035 (ops 200986-206875) I0108 16:40:02.811221 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000036 (ops 206876-212809) I0108 16:40:02.957545 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.147s user 0.002s sys 0.029s Metrics: {} I0108 16:40:02.959177 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 148099023 bytes on disk I0108 16:40:03.276489 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.317s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":316930,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":4,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":8} I0108 16:40:03.303107 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 52.136s user 9.653s sys 1.000s Metrics: {"bytes_written":124022961,"cfile_cache_hit":373,"cfile_cache_hit_bytes":52349745,"cfile_cache_miss":16397,"cfile_cache_miss_bytes":451679129,"cfile_init":23,"delta_iterators_relevant":14,"dirs.queue_time_us":15925,"dirs.run_cpu_time_us":451,"dirs.run_wall_time_us":229142,"drs_written":4,"lbm_read_time_us":4403888,"lbm_reads_1-10_ms":435,"lbm_reads_10-100_ms":51,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":15993,"lbm_write_time_us":1837872,"lbm_writes_1-10_ms":410,"lbm_writes_10-100_ms":18,"lbm_writes_lt_1ms":12175,"num_input_rowsets":5,"rows_written":7112515,"spinlock_wait_cycles":642439424,"thread_start_us":638,"threads_started":12} I0108 16:40:03.303325 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 148537985 bytes on disk I0108 16:40:03.410704 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 51.956s user 9.710s sys 0.988s Metrics: {"bytes_written":124175611,"cfile_cache_hit":399,"cfile_cache_hit_bytes":56289223,"cfile_cache_miss":16393,"cfile_cache_miss_bytes":448382291,"cfile_init":23,"delta_iterators_relevant":14,"dirs.queue_time_us":35889,"dirs.run_cpu_time_us":554,"dirs.run_wall_time_us":189735,"drs_written":4,"lbm_read_time_us":4780976,"lbm_reads_1-10_ms":372,"lbm_reads_10-100_ms":67,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":16038,"lbm_write_time_us":1795559,"lbm_writes_1-10_ms":422,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":12186,"num_input_rowsets":5,"rows_written":7121349,"spinlock_wait_cycles":558396672,"thread_start_us":681,"threads_started":12} I0108 16:40:03.410822 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 100958941 bytes of WAL I0108 16:40:03.410889 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 12 log segments from log reader I0108 16:40:03.410900 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000025 (ops 141752-147692) I0108 16:40:03.410935 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000026 (ops 147693-153569) I0108 16:40:03.410956 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000027 (ops 153570-159544) I0108 16:40:03.410974 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000028 (ops 159545-165462) I0108 16:40:03.410990 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000029 (ops 165463-171369) I0108 16:40:03.411006 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000030 (ops 171370-177306) I0108 16:40:03.411025 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000031 (ops 177307-183099) I0108 16:40:03.411041 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000032 (ops 183100-188956) I0108 16:40:03.411057 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000033 (ops 188957-194944) I0108 16:40:03.411072 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000034 (ops 194945-200764) I0108 16:40:03.411089 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000035 (ops 200765-206756) I0108 16:40:03.411108 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000036 (ops 206757-212669) I0108 16:40:03.441956 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.031s user 0.000s sys 0.031s Metrics: {} I0108 16:40:03.442078 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 148727786 bytes on disk I0108 16:40:03.461657 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.158s user 0.001s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":158047,"lbm_reads_1-10_ms":6,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":8} I0108 16:40:03.620918 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.179s user 0.000s sys 0.000s Metrics: {"cfile_init":4,"lbm_read_time_us":178623,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":8} I0108 16:40:04.973948 26729 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=734565, attempt_no=0}) took 1082ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":430,"cfile_cache_hit":36,"cfile_cache_hit_bytes":189463,"cfile_cache_miss":1065,"cfile_cache_miss_bytes":4372890,"lbm_read_time_us":1069085,"lbm_reads_1-10_ms":42,"lbm_reads_10-100_ms":28,"lbm_reads_lt_1ms":995,"num_ops":1256,"prepare.queue_time_us":13,"prepare.run_cpu_time_us":1089,"prepare.run_wall_time_us":4163,"raft.queue_time_us":79,"raft.run_cpu_time_us":37,"raft.run_wall_time_us":37,"replication_time_us":508}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 250000 Mean: 1780.53 I0108 16:40:09.041286 24342 linked_list-test-util.h:519] Written 12500000 rows in chain Percentiles: 0% (min) = 728 25% = 1337 50% (med) = 1500 75% = 1707 95% = 2276 99% = 7212 99.9% = 44064 99.99% = 84672 100% (max) = 643737 I0108 16:40:12.377940 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033380 I0108 16:40:12.468631 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033351 I0108 16:40:13.035524 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033334 I0108 16:40:15.774719 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 3.397s user 0.724s sys 0.064s Metrics: {"bytes_written":8672616,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":287566,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":131122,"lbm_writes_1-10_ms":38,"lbm_writes_lt_1ms":846,"reinsert_count":0,"rows_written":490400,"thread_start_us":30,"threads_started":1,"update_count":19} I0108 16:40:15.774899 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 33602621 bytes of WAL I0108 16:40:15.774972 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 4 log segments from log reader I0108 16:40:15.775127 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000016 (ops 186655-199158) I0108 16:40:15.776012 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000017 (ops 199159-211594) I0108 16:40:15.777228 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000018 (ops 211595-224103) I0108 16:40:15.778053 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000019 (ops 224104-236542) I0108 16:40:15.849717 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.075s user 0.000s sys 0.013s Metrics: {} I0108 16:40:15.849874 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.001055 I0108 16:40:15.892642 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 3.424s user 0.726s sys 0.061s Metrics: {"bytes_written":8689077,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":16088,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":105325,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":186342,"lbm_writes_1-10_ms":40,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":843,"reinsert_count":0,"rows_written":490929,"thread_start_us":32,"threads_started":1,"update_count":330} I0108 16:40:15.892777 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 33656680 bytes of WAL I0108 16:40:15.892819 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 4 log segments from log reader I0108 16:40:15.892827 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000016 (ops 187237-199698) I0108 16:40:15.892868 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000017 (ops 199699-212184) I0108 16:40:15.892889 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000018 (ops 212185-224713) I0108 16:40:15.892906 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000019 (ops 224714-237243) I0108 16:40:15.900923 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.008s user 0.001s sys 0.007s Metrics: {} I0108 16:40:15.901039 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.001049 I0108 16:40:16.053784 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 3.018s user 0.700s sys 0.066s Metrics: {"bytes_written":8709138,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":52349,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":121952,"lbm_writes_1-10_ms":21,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":861,"reinsert_count":0,"rows_written":490420,"thread_start_us":34,"threads_started":1,"update_count":3035} I0108 16:40:16.054320 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 33589176 bytes of WAL I0108 16:40:16.054401 24439 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 4 log segments from log reader I0108 16:40:16.054474 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000016 (ops 186055-198440) I0108 16:40:16.054556 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000017 (ops 198441-210857) I0108 16:40:16.054616 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000018 (ops 210858-223279) I0108 16:40:16.054672 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000019 (ops 223280-235648) I0108 16:40:16.066659 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.012s user 0.000s sys 0.010s Metrics: {} I0108 16:40:16.066807 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.001052 I0108 16:40:30.021926 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 14.121s user 2.825s sys 0.293s Metrics: {"bytes_written":36748600,"cfile_cache_hit":97,"cfile_cache_hit_bytes":12847677,"cfile_cache_miss":4853,"cfile_cache_miss_bytes":136628061,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":408,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":260505,"drs_written":2,"lbm_read_time_us":1959207,"lbm_reads_1-10_ms":107,"lbm_reads_10-100_ms":36,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":4747,"lbm_write_time_us":416567,"lbm_writes_1-10_ms":93,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":3637,"num_input_rowsets":2,"rows_written":2097861,"spinlock_wait_cycles":57278976,"thread_start_us":587,"threads_started":10} I0108 16:40:30.023417 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034670 I0108 16:40:30.160863 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 14.311s user 2.827s sys 0.308s Metrics: {"bytes_written":36762408,"cfile_cache_hit":98,"cfile_cache_hit_bytes":12992981,"cfile_cache_miss":4854,"cfile_cache_miss_bytes":136566833,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":5904,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":108169,"drs_written":2,"lbm_read_time_us":1817300,"lbm_reads_1-10_ms":112,"lbm_reads_10-100_ms":31,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":4750,"lbm_write_time_us":516895,"lbm_writes_1-10_ms":85,"lbm_writes_10-100_ms":11,"lbm_writes_lt_1ms":3640,"num_input_rowsets":2,"rows_written":2098893,"spinlock_wait_cycles":73772160,"thread_start_us":284,"threads_started":10} I0108 16:40:30.160981 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 44083708 bytes on disk I0108 16:40:30.164490 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":8} I0108 16:40:30.164562 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.034358 I0108 16:40:30.270967 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 14.204s user 2.797s sys 0.275s Metrics: {"bytes_written":36801936,"cfile_cache_hit":82,"cfile_cache_hit_bytes":10579511,"cfile_cache_miss":4874,"cfile_cache_miss_bytes":139123243,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":3721,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":122275,"drs_written":2,"lbm_read_time_us":2229093,"lbm_reads_1-10_ms":115,"lbm_reads_10-100_ms":40,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":4753,"lbm_write_time_us":511327,"lbm_writes_1-10_ms":94,"lbm_writes_10-100_ms":2,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3641,"num_input_rowsets":2,"rows_written":2100566,"spinlock_wait_cycles":6914304,"thread_start_us":519,"threads_started":10} I0108 16:40:30.271163 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 44122519 bytes on disk I0108 16:40:30.271281 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":13,"lbm_reads_lt_1ms":8} I0108 16:40:30.271404 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034233 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 260000 Mean: 1789.29 Percentiles: 0% (min) = 728 25% = 1338 50% (med) = 1501 75% = 1710 95% = 2280 99% = 7284 99.9% = 45376 99.99% = 90240 100% (max) = 643737 I0108 16:40:30.717563 24342 linked_list-test-util.h:519] Written 13000000 rows in chain I0108 16:40:32.559900 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 2.395s user 0.668s sys 0.048s Metrics: {"bytes_written":8857729,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":22618,"dirs.run_cpu_time_us":92,"dirs.run_wall_time_us":37807,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":238629,"lbm_writes_1-10_ms":26,"lbm_writes_10-100_ms":8,"lbm_writes_lt_1ms":870,"reinsert_count":0,"rows_written":500604,"thread_start_us":30,"threads_started":1,"update_count":112} I0108 16:40:32.560317 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 41928260 bytes of WAL I0108 16:40:32.561206 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:40:32.561259 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000016 (ops 186129-198440) I0108 16:40:32.562491 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000017 (ops 198441-210862) I0108 16:40:32.562566 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000018 (ops 210863-223283) I0108 16:40:32.562650 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000019 (ops 223284-235658) I0108 16:40:32.562803 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000020 (ops 235659-248029) I0108 16:40:32.694386 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.133s user 0.000s sys 0.017s Metrics: {} I0108 16:40:32.694648 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.000956 I0108 16:40:32.770752 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 2.747s user 0.731s sys 0.050s Metrics: {"bytes_written":8884028,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":18339,"dirs.run_cpu_time_us":102,"dirs.run_wall_time_us":51188,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":384669,"lbm_writes_1-10_ms":41,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":857,"reinsert_count":0,"rows_written":502050,"thread_start_us":29,"threads_started":1,"update_count":26} I0108 16:40:32.770865 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41954695 bytes of WAL I0108 16:40:32.770921 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:40:32.770936 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000016 (ops 186026-198339) I0108 16:40:32.770967 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000017 (ops 198340-210746) I0108 16:40:32.770994 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000018 (ops 210747-223168) I0108 16:40:32.771014 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000019 (ops 223169-235561) I0108 16:40:32.771034 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000020 (ops 235562-247932) I0108 16:40:32.780766 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.010s user 0.000s sys 0.009s Metrics: {} I0108 16:40:32.780886 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.000940 I0108 16:40:33.073993 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 2.802s user 0.723s sys 0.054s Metrics: {"bytes_written":8783993,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":1278,"dirs.run_cpu_time_us":109,"dirs.run_wall_time_us":40953,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":260041,"lbm_writes_1-10_ms":31,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":861,"reinsert_count":0,"rows_written":496174,"thread_start_us":116,"threads_started":1,"update_count":169} I0108 16:40:33.074362 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 9853522 bytes on disk I0108 16:40:33.074527 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 16:40:33.074584 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.000999 I0108 16:40:47.091380 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 14.310s user 2.890s sys 0.312s Metrics: {"bytes_written":38131624,"cfile_cache_hit":90,"cfile_cache_hit_bytes":11716066,"cfile_cache_miss":5022,"cfile_cache_miss_bytes":142814418,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":21313,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":73546,"drs_written":2,"lbm_read_time_us":1801963,"lbm_reads_1-10_ms":103,"lbm_reads_10-100_ms":23,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":4930,"lbm_write_time_us":576876,"lbm_writes_1-10_ms":113,"lbm_writes_10-100_ms":3,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3739,"num_input_rowsets":2,"rows_written":2167254,"spinlock_wait_cycles":24877184,"thread_start_us":583,"threads_started":10} I0108 16:40:47.092507 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 45589785 bytes on disk I0108 16:40:47.094427 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0108 16:40:47.094535 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 44063890 bytes on disk I0108 16:40:47.176709 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.082s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":81977,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":3} I0108 16:40:47.696393 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 15.002s user 2.881s sys 0.290s Metrics: {"bytes_written":38109900,"cfile_cache_hit":99,"cfile_cache_hit_bytes":13072856,"cfile_cache_miss":5009,"cfile_cache_miss_bytes":141351394,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":26182,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":60616,"drs_written":2,"lbm_read_time_us":1293378,"lbm_reads_1-10_ms":119,"lbm_reads_10-100_ms":25,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":4903,"lbm_write_time_us":522627,"lbm_writes_1-10_ms":130,"lbm_writes_10-100_ms":6,"lbm_writes_lt_1ms":3718,"num_input_rowsets":2,"rows_written":2165897,"spinlock_wait_cycles":10647552,"thread_start_us":486,"threads_started":10} I0108 16:40:47.696547 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 45566282 bytes on disk I0108 16:40:47.696710 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.001s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} I0108 16:40:47.776684 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 14.702s user 2.890s sys 0.324s Metrics: {"bytes_written":38133495,"cfile_cache_hit":99,"cfile_cache_hit_bytes":13135823,"cfile_cache_miss":5019,"cfile_cache_miss_bytes":141422049,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":397,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":151709,"drs_written":2,"lbm_read_time_us":1553810,"lbm_reads_1-10_ms":115,"lbm_reads_10-100_ms":26,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":4911,"lbm_write_time_us":481080,"lbm_writes_1-10_ms":105,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":3746,"num_input_rowsets":2,"rows_written":2167397,"spinlock_wait_cycles":43266944,"thread_start_us":707,"threads_started":10} I0108 16:40:47.776793 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41998392 bytes of WAL I0108 16:40:47.776842 24439 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:40:47.776854 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000016 (ops 187144-199669) I0108 16:40:47.776901 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000017 (ops 199670-212089) I0108 16:40:47.776923 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000018 (ops 212090-224586) I0108 16:40:47.776938 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000019 (ops 224587-237116) I0108 16:40:47.776953 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000020 (ops 237117-249596) I0108 16:40:47.800158 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.023s user 0.000s sys 0.015s Metrics: {} I0108 16:40:47.800303 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 45592698 bytes on disk I0108 16:40:47.914599 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.114s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":113959,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":4} I0108 16:40:51.664083 24342 linked_list-test-util.h:519] Written 13500000 rows in chain ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 270000 Mean: 1795.37 Percentiles: 0% (min) = 728 25% = 1339 50% (med) = 1503 75% = 1712 95% = 2290 99% = 7404 99.9% = 45984 99.99% = 91776 100% (max) = 643737 I0108 16:41:08.932893 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033391 I0108 16:41:11.206084 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033399 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 280000 Mean: 1795.08 Percentiles: 0% (min) = 728 25% = 1340 50% (med) = 1504 75% = 1712 95% = 2286 99% = 7412 99.9% = 45696 99.99% = 90240 100% (max) = 643737 I0108 16:41:11.401594 24342 linked_list-test-util.h:519] Written 14000000 rows in chain I0108 16:41:11.465644 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033337 I0108 16:41:24.945554 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 16.013s user 3.356s sys 0.286s Metrics: {"bytes_written":38914453,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":9935,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":132163,"drs_written":2,"lbm_read_time_us":49,"lbm_reads_lt_1ms":8,"lbm_write_time_us":536378,"lbm_writes_1-10_ms":110,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":3803,"reinsert_count":0,"rows_written":2197739,"thread_start_us":66,"threads_started":2,"update_count":7327} I0108 16:41:24.967983 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 92937809 bytes of WAL I0108 16:41:24.968104 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 11 log segments from log reader I0108 16:41:24.968147 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000037 (ops 212810-218686) I0108 16:41:24.968269 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000038 (ops 218687-224608) I0108 16:41:24.968336 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000039 (ops 224609-230542) I0108 16:41:24.968396 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000040 (ops 230543-236426) I0108 16:41:24.968461 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000041 (ops 236427-242406) I0108 16:41:24.968552 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000042 (ops 242407-248346) I0108 16:41:24.968616 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000043 (ops 248347-254289) I0108 16:41:24.968706 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000044 (ops 254290-260221) I0108 16:41:24.968770 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000045 (ops 260222-266359) I0108 16:41:24.968829 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000046 (ops 266360-272291) I0108 16:41:24.968888 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000047 (ops 272292-278135) I0108 16:41:25.047161 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.079s user 0.001s sys 0.027s Metrics: {} I0108 16:41:25.053791 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 43794651 bytes on disk I0108 16:41:25.053958 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8} I0108 16:41:25.054040 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.862499 I0108 16:41:29.046357 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 17.840s user 3.549s sys 0.290s Metrics: {"bytes_written":39273987,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":30884,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":118398,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":691870,"lbm_writes_1-10_ms":135,"lbm_writes_10-100_ms":3,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":3808,"reinsert_count":0,"rows_written":2216077,"thread_start_us":59,"threads_started":2,"update_count":11362} I0108 16:41:29.049360 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 92507784 bytes of WAL I0108 16:41:29.049698 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 11 log segments from log reader I0108 16:41:29.049751 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000037 (ops 213698-219647) I0108 16:41:29.049839 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000038 (ops 219648-225621) I0108 16:41:29.050068 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000039 (ops 225622-231562) I0108 16:41:29.050159 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000040 (ops 231563-237515) I0108 16:41:29.050226 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000041 (ops 237516-243492) I0108 16:41:29.050293 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000042 (ops 243493-249472) I0108 16:41:29.050362 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000043 (ops 249473-255348) I0108 16:41:29.050495 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000044 (ops 255349-261266) I0108 16:41:29.050559 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000045 (ops 261267-267319) I0108 16:41:29.050617 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000046 (ops 267320-273170) I0108 16:41:29.050668 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000047 (ops 273171-279064) I0108 16:41:29.211936 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.162s user 0.000s sys 0.028s Metrics: {} I0108 16:41:29.212105 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 44148869 bytes on disk I0108 16:41:29.212256 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} I0108 16:41:29.212558 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.856116 I0108 16:41:29.406108 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 17.940s user 3.603s sys 0.292s Metrics: {"bytes_written":39252520,"cfile_init":2,"delete_count":0,"dirs.queue_time_us":13711,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":87449,"drs_written":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8,"lbm_write_time_us":517469,"lbm_writes_1-10_ms":135,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3804,"reinsert_count":0,"rows_written":2212091,"thread_start_us":74,"threads_started":2,"update_count":16210} I0108 16:41:29.406350 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 92698461 bytes of WAL I0108 16:41:29.407577 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 11 log segments from log reader I0108 16:41:29.407596 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000037 (ops 212670-218675) I0108 16:41:29.407645 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000038 (ops 218676-224561) I0108 16:41:29.407670 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000039 (ops 224562-230542) I0108 16:41:29.407691 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000040 (ops 230543-236420) I0108 16:41:29.407712 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000041 (ops 236421-242396) I0108 16:41:29.407734 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000042 (ops 242397-248347) I0108 16:41:29.407753 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000043 (ops 248348-254289) I0108 16:41:29.407810 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000044 (ops 254290-260161) I0108 16:41:29.407902 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000045 (ops 260162-266120) I0108 16:41:29.407923 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000046 (ops 266121-271928) I0108 16:41:29.409158 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000047 (ops 271929-277899) I0108 16:41:29.624681 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.217s user 0.000s sys 0.031s Metrics: {} I0108 16:41:29.626361 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 44034482 bytes on disk I0108 16:41:29.626531 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":19,"lbm_reads_lt_1ms":8} I0108 16:41:29.626615 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.857669 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 290000 Mean: 1800.86 Percentiles: 0% (min) = 728 25% = 1343 50% (med) = 1506 75% = 1715 95% = 2296 99% = 7476 99.9% = 46368 99.99% = 90240 100% (max) = 643737 I0108 16:41:32.408087 24342 linked_list-test-util.h:519] Written 14500000 rows in chain W0108 16:41:47.281946 24684 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.23.197.130:37520, user_credentials={real_user=jenkins}} blocked reactor thread for 59658.6us ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. I0108 16:41:54.640707 24342 linked_list-test-util.h:519] Written 15000000 rows in chain ------------------------------------------------------------ Count: 300000 Mean: 1808.88 Percentiles: 0% (min) = 728 25% = 1346 50% (med) = 1511 75% = 1721 95% = 2314 99% = 7556 99.9% = 46752 99.99% = 87616 100% (max) = 643737 I0108 16:42:00.349884 24342 linked_list-test-util.h:528] Sample at HT timestamp: 6465661011347656704 Inserted count: 15130600 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 310000 Mean: 1814.91 Percentiles: 0% (min) = 728 25% = 1349 50% (med) = 1514 75% = 1726 95% = 2326 99% = 7640 99.9% = 46816 99.99% = 87104 100% (max) = 643737 I0108 16:42:16.329282 24342 linked_list-test-util.h:519] Written 15500000 rows in chain I0108 16:42:34.304046 24848 thread.cc:387] Could not get per-thread statistics: Not found: /proc/self/task/27094/stat: No such file or directory (error 2) I0108 16:42:36.258747 26957 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=944058, attempt_no=0}) took 1010ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":9,"cfile_cache_hit":123301,"cfile_cache_hit_bytes":1282490149,"num_ops":14699,"prepare.queue_time_us":9,"prepare.run_cpu_time_us":15712,"prepare.run_wall_time_us":49840,"raft.queue_time_us":53,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":164,"replication_time_us":2624,"spinlock_wait_cycles":4227456}]]} I0108 16:42:37.319023 26957 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=945322, attempt_no=0}) took 1053ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":15,"cfile_cache_hit":118472,"cfile_cache_hit_bytes":1232123866,"num_ops":14114,"prepare.queue_time_us":142,"prepare.run_cpu_time_us":15406,"prepare.run_wall_time_us":58277,"raft.queue_time_us":24,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":130,"replication_time_us":6439,"spinlock_wait_cycles":4474752}]]} I0108 16:42:38.276939 27008 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=946611, attempt_no=0}) took 1005ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":23,"cfile_cache_hit":121522,"cfile_cache_hit_bytes":1263025898,"num_ops":14474,"prepare.queue_time_us":13,"prepare.run_cpu_time_us":17319,"prepare.run_wall_time_us":65583,"raft.queue_time_us":5002,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":148,"replication_time_us":14457,"spinlock_wait_cycles":1741056}]]} ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 320000 Mean: 1823.61 Percentiles: 0% (min) = 728 25% = 1352 50% (med) = 1519 75% = 1733 95% = 2344 99% = 7716 99.9% = 47136 99.99% = 87104 100% (max) = 643737 I0108 16:42:38.631536 24342 linked_list-test-util.h:519] Written 16000000 rows in chain I0108 16:42:41.403961 26957 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=950393, attempt_no=0}) took 1046ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":62,"cfile_cache_hit":126194,"cfile_cache_hit_bytes":1311424919,"num_ops":15028,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":16120,"prepare.run_wall_time_us":56347,"raft.queue_time_us":996,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":132,"replication_time_us":2606,"spinlock_wait_cycles":18430336}]]} I0108 16:42:42.233610 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 77.179s user 12.339s sys 1.236s Metrics: {"bytes_written":158251339,"cfile_cache_hit":412,"cfile_cache_hit_bytes":57372544,"cfile_cache_miss":20786,"cfile_cache_miss_bytes":585918280,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":54714,"dirs.run_cpu_time_us":528,"dirs.run_wall_time_us":288539,"drs_written":5,"lbm_read_time_us":6059806,"lbm_reads_1-10_ms":623,"lbm_reads_10-100_ms":95,"lbm_reads_gt_100_ms":12,"lbm_reads_lt_1ms":20140,"lbm_write_time_us":2837428,"lbm_writes_1-10_ms":563,"lbm_writes_10-100_ms":26,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":15344,"num_input_rowsets":5,"rows_written":8993615,"spinlock_wait_cycles":1085514496,"thread_start_us":161,"threads_started":5} I0108 16:42:42.233747 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.145681 I0108 16:42:42.375023 27027 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=951655, attempt_no=0}) took 1010ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":3558,"cfile_cache_hit":119449,"cfile_cache_hit_bytes":1241494319,"num_ops":14216,"prepare.queue_time_us":15,"prepare.run_cpu_time_us":15576,"prepare.run_wall_time_us":57534,"raft.queue_time_us":22,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":140,"replication_time_us":11081,"spinlock_wait_cycles":2111488}]]} I0108 16:42:43.080207 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 73.868s user 12.390s sys 1.216s Metrics: {"bytes_written":158627237,"cfile_cache_hit":430,"cfile_cache_hit_bytes":60002568,"cfile_cache_miss":20822,"cfile_cache_miss_bytes":584936060,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":54842,"dirs.run_cpu_time_us":574,"dirs.run_wall_time_us":260025,"drs_written":5,"lbm_read_time_us":6163870,"lbm_reads_1-10_ms":472,"lbm_reads_10-100_ms":107,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":20318,"lbm_write_time_us":2568050,"lbm_writes_1-10_ms":562,"lbm_writes_10-100_ms":18,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":15393,"num_input_rowsets":5,"rows_written":9015753,"spinlock_wait_cycles":1297502208,"thread_start_us":555,"threads_started":13} I0108 16:42:43.080473 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.151504 I0108 16:42:43.563750 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 73.937s user 12.333s sys 1.213s Metrics: {"bytes_written":158784121,"cfile_cache_hit":455,"cfile_cache_hit_bytes":63665457,"cfile_cache_miss":20817,"cfile_cache_miss_bytes":581944805,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":88913,"dirs.run_cpu_time_us":486,"dirs.run_wall_time_us":127524,"drs_written":5,"lbm_read_time_us":6532235,"lbm_reads_1-10_ms":486,"lbm_reads_10-100_ms":106,"lbm_reads_gt_100_ms":12,"lbm_reads_lt_1ms":20297,"lbm_write_time_us":2363910,"lbm_writes_1-10_ms":576,"lbm_writes_10-100_ms":21,"lbm_writes_lt_1ms":15389,"num_input_rowsets":5,"rows_written":9024579,"spinlock_wait_cycles":856190336,"thread_start_us":844,"threads_started":13} I0108 16:42:43.563915 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 190856298 bytes on disk I0108 16:42:43.924273 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.360s user 0.001s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":360051,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":8,"lbm_reads_lt_1ms":10} I0108 16:42:43.924417 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.150345 I0108 16:42:45.340013 27189 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=952863, attempt_no=0}) took 1587ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":21,"cfile_cache_hit":10,"cfile_cache_hit_bytes":51149,"cfile_cache_miss":39,"cfile_cache_miss_bytes":201871,"cfile_init":4,"lbm_read_time_us":1585035,"lbm_reads_1-10_ms":8,"lbm_reads_10-100_ms":38,"lbm_reads_lt_1ms":9,"num_ops":35,"prepare.queue_time_us":80,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":110,"raft.queue_time_us":76,"raft.run_cpu_time_us":34,"raft.run_wall_time_us":34,"replication_time_us":842,"spinlock_wait_cycles":6016}]]} I0108 16:42:46.410763 27189 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=952869, attempt_no=0}) took 1070ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":55,"cfile_cache_hit":12,"cfile_cache_hit_bytes":101098,"cfile_cache_miss":34,"cfile_cache_miss_bytes":139604,"lbm_read_time_us":1068075,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":29,"lbm_reads_lt_1ms":2,"num_ops":35,"prepare.queue_time_us":165,"prepare.run_cpu_time_us":189,"prepare.run_wall_time_us":653,"raft.queue_time_us":192,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":236,"replication_time_us":809,"spinlock_wait_cycles":474496,"thread_start_us":233,"threads_started":2}]]} I0108 16:42:48.536921 27189 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=952881, attempt_no=0}) took 1005ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":443,"cfile_cache_hit":13,"cfile_cache_hit_bytes":105204,"cfile_cache_miss":29,"cfile_cache_miss_bytes":119074,"lbm_read_time_us":1003744,"lbm_reads_1-10_ms":5,"lbm_reads_10-100_ms":12,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":9,"num_ops":32,"prepare.queue_time_us":11,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":107,"raft.queue_time_us":49,"raft.run_cpu_time_us":33,"raft.run_wall_time_us":31,"replication_time_us":500}]]} I0108 16:42:49.545450 27350 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=952886, attempt_no=0}) took 1008ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":89,"cfile_cache_hit":13,"cfile_cache_hit_bytes":105204,"cfile_cache_miss":27,"cfile_cache_miss_bytes":110862,"lbm_read_time_us":1006707,"lbm_reads_1-10_ms":8,"lbm_reads_10-100_ms":11,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":6,"num_ops":31,"prepare.queue_time_us":25,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":98,"raft.queue_time_us":46,"raft.run_cpu_time_us":30,"raft.run_wall_time_us":30,"replication_time_us":501}]]} I0108 16:42:49.804255 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 7.570s user 1.575s sys 0.126s Metrics: {"bytes_written":27418345,"cfile_cache_hit":31,"cfile_cache_hit_bytes":2901792,"cfile_cache_miss":3649,"cfile_cache_miss_bytes":108750632,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":144,"dirs.run_cpu_time_us":84,"dirs.run_wall_time_us":84,"drs_written":1,"lbm_read_time_us":4268753,"lbm_reads_1-10_ms":30,"lbm_reads_10-100_ms":110,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3544,"lbm_write_time_us":1173839,"lbm_writes_1-10_ms":24,"lbm_writes_10-100_ms":44,"lbm_writes_lt_1ms":2692,"num_input_rowsets":2,"rows_written":1556403,"spinlock_wait_cycles":384,"thread_start_us":457,"threads_started":9} I0108 16:42:49.804497 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 196725093 bytes on disk I0108 16:42:49.990490 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.186s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":185665,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":6,"lbm_reads_lt_1ms":10} I0108 16:42:49.990594 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.042760 I0108 16:42:50.989831 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.999s user 0.574s sys 0.025s Metrics: {"bytes_written":10167532,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":85793,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":271951,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":1035,"reinsert_count":0,"rows_written":579646,"thread_start_us":33,"threads_started":1,"update_count":18} I0108 16:42:50.989944 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 50355315 bytes of WAL I0108 16:42:50.990021 24439 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 6 log segments from log reader I0108 16:42:50.990041 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000020 (ops 235649-248011) I0108 16:42:50.990097 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000021 (ops 248012-260446) I0108 16:42:50.990128 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000022 (ops 260447-272750) I0108 16:42:50.990146 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000023 (ops 272751-285205) I0108 16:42:50.990164 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000024 (ops 285206-297561) I0108 16:42:50.990192 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000025 (ops 297562-309938) I0108 16:42:51.002894 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.013s user 0.000s sys 0.011s Metrics: {} I0108 16:42:51.003019 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 11578596 bytes on disk I0108 16:42:51.003141 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0108 16:42:51.003213 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.007438 I0108 16:42:51.303699 27027 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=952858, attempt_no=0}) took 7729ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":14,"cfile_cache_hit":71,"cfile_cache_hit_bytes":331550,"cfile_cache_miss":543,"cfile_cache_miss_bytes":2241360,"cfile_init":1,"lbm_read_time_us":7720349,"lbm_reads_1-10_ms":83,"lbm_reads_10-100_ms":165,"lbm_reads_gt_100_ms":16,"lbm_reads_lt_1ms":283,"num_ops":590,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":670,"raft.queue_time_us":272,"raft.run_cpu_time_us":33,"raft.run_wall_time_us":33,"replication_time_us":1049}]]} W0108 16:42:55.412183 24689 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.316s user 0.001s sys 0.000s I0108 16:42:55.413653 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 12.333s user 1.591s sys 0.134s Metrics: {"bytes_written":28098249,"cfile_cache_hit":31,"cfile_cache_hit_bytes":2900708,"cfile_cache_miss":3739,"cfile_cache_miss_bytes":111510634,"cfile_init":9,"delta_iterators_relevant":6,"dirs.queue_time_us":447,"dirs.run_cpu_time_us":131,"dirs.run_wall_time_us":80825,"drs_written":1,"lbm_read_time_us":5129544,"lbm_reads_1-10_ms":21,"lbm_reads_10-100_ms":115,"lbm_reads_gt_100_ms":7,"lbm_reads_lt_1ms":3632,"lbm_write_time_us":3768775,"lbm_writes_1-10_ms":43,"lbm_writes_10-100_ms":94,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":2687,"num_input_rowsets":2,"rows_written":1595272,"thread_start_us":650,"threads_started":9} I0108 16:42:55.413794 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.044311 I0108 16:42:58.189935 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 2.776s user 0.631s sys 0.028s Metrics: {"bytes_written":10217415,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":81,"dirs.run_cpu_time_us":107,"dirs.run_wall_time_us":71029,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1976374,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":48,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":996,"reinsert_count":0,"rows_written":582434,"thread_start_us":40,"threads_started":1,"update_count":16} I0108 16:42:58.190063 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 50416150 bytes of WAL I0108 16:42:58.190130 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 6 log segments from log reader I0108 16:42:58.190150 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000020 (ops 237244-249703) I0108 16:42:58.190209 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000021 (ops 249704-262186) I0108 16:42:58.190243 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000022 (ops 262187-274616) I0108 16:42:58.190263 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000023 (ops 274617-287117) I0108 16:42:58.190295 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000024 (ops 287118-299597) I0108 16:42:58.190356 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000025 (ops 299598-312021) I0108 16:42:58.202143 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.012s user 0.000s sys 0.011s Metrics: {} I0108 16:42:58.202265 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.007438 I0108 16:42:59.977670 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 16.053s user 1.555s sys 0.133s Metrics: {"bytes_written":28181657,"cfile_cache_hit":33,"cfile_cache_hit_bytes":3037706,"cfile_cache_miss":3747,"cfile_cache_miss_bytes":111744944,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":328,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":83137,"drs_written":1,"lbm_read_time_us":5395081,"lbm_reads_1-10_ms":18,"lbm_reads_10-100_ms":119,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":3634,"lbm_write_time_us":7620803,"lbm_writes_1-10_ms":41,"lbm_writes_10-100_ms":166,"lbm_writes_gt_100_ms":9,"lbm_writes_lt_1ms":2619,"num_input_rowsets":2,"rows_written":1600033,"spinlock_wait_cycles":768,"thread_start_us":576,"threads_started":9} I0108 16:42:59.977843 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 33891905 bytes on disk I0108 16:43:00.110255 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.132s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132254,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":1} I0108 16:43:00.110414 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.045649 I0108 16:43:01.427525 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 1.317s user 0.606s sys 0.028s Metrics: {"bytes_written":10205446,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":66376,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":567451,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":12,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":1032,"reinsert_count":0,"rows_written":581555,"thread_start_us":30,"threads_started":1,"update_count":18} I0108 16:43:01.427646 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 50349728 bytes of WAL I0108 16:43:01.427713 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 6 log segments from log reader I0108 16:43:01.427728 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000020 (ops 236543-248956) I0108 16:43:01.427786 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000021 (ops 248957-261484) I0108 16:43:01.427815 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000022 (ops 261485-273945) I0108 16:43:01.427835 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000023 (ops 273946-286391) I0108 16:43:01.427865 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000024 (ops 286392-298841) I0108 16:43:01.427883 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000025 (ops 298842-311316) I0108 16:43:01.439674 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.012s user 0.000s sys 0.012s Metrics: {} I0108 16:43:01.439791 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.007434 I0108 16:43:01.764571 27577 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=952989, attempt_no=0}) took 1225ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":245,"cfile_cache_hit":10,"cfile_cache_hit_bytes":97241,"cfile_cache_miss":28,"cfile_cache_miss_bytes":114968,"lbm_read_time_us":1223438,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":21,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":2,"num_ops":33,"prepare.queue_time_us":274,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":467,"raft.queue_time_us":23,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":63,"replication_time_us":817,"spinlock_wait_cycles":64768,"thread_start_us":156,"threads_started":2}]]} I0108 16:43:06.404032 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 15.401s user 2.586s sys 0.182s Metrics: {"bytes_written":47147563,"cfile_cache_hit":114,"cfile_cache_hit_bytes":14461600,"cfile_cache_miss":6216,"cfile_cache_miss_bytes":177954424,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":598,"dirs.run_cpu_time_us":295,"dirs.run_wall_time_us":156182,"drs_written":2,"lbm_read_time_us":7516911,"lbm_reads_1-10_ms":34,"lbm_reads_10-100_ms":186,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":6042,"lbm_write_time_us":4424491,"lbm_writes_1-10_ms":31,"lbm_writes_10-100_ms":85,"lbm_writes_gt_100_ms":8,"lbm_writes_lt_1ms":4632,"num_input_rowsets":3,"rows_written":2680212,"thread_start_us":682,"threads_started":10} I0108 16:43:06.404219 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 57089215 bytes on disk I0108 16:43:06.524102 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.120s user 0.001s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":119678,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":4} I0108 16:43:06.524246 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.042625 I0108 16:43:07.606170 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 1.082s user 0.511s sys 0.020s Metrics: {"bytes_written":9020004,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":50679,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":436883,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":910,"rows_written":513128,"thread_start_us":35,"threads_started":1} I0108 16:43:07.606312 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41985211 bytes of WAL I0108 16:43:07.606360 24439 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:43:07.606370 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000021 (ops 249597-262089) I0108 16:43:07.606398 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000022 (ops 262090-274547) I0108 16:43:07.606421 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000023 (ops 274548-286969) I0108 16:43:07.606462 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000024 (ops 286970-299432) I0108 16:43:07.606484 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000025 (ops 299433-311893) I0108 16:43:07.617512 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.011s user 0.000s sys 0.011s Metrics: {} I0108 16:43:07.617594 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.007409 I0108 16:43:07.700291 27669 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=953031, attempt_no=0}) took 1156ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":9,"cfile_cache_hit":10,"cfile_cache_hit_bytes":97241,"cfile_cache_miss":30,"cfile_cache_miss_bytes":123180,"lbm_read_time_us":1154880,"lbm_reads_1-10_ms":8,"lbm_reads_10-100_ms":20,"lbm_reads_gt_100_ms":2,"num_ops":35,"prepare.queue_time_us":11,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":103,"raft.queue_time_us":54,"raft.run_cpu_time_us":33,"raft.run_wall_time_us":32,"replication_time_us":725}]]} I0108 16:43:14.532456 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 16.330s user 2.668s sys 0.201s Metrics: {"bytes_written":47149356,"cfile_cache_hit":108,"cfile_cache_hit_bytes":13565197,"cfile_cache_miss":6222,"cfile_cache_miss_bytes":178827651,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":435,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":96210,"drs_written":2,"lbm_read_time_us":7056523,"lbm_reads_1-10_ms":44,"lbm_reads_10-100_ms":139,"lbm_reads_gt_100_ms":13,"lbm_reads_lt_1ms":6078,"lbm_write_time_us":5846401,"lbm_writes_1-10_ms":74,"lbm_writes_10-100_ms":129,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":4550,"num_input_rowsets":3,"rows_written":2680295,"spinlock_wait_cycles":384,"thread_start_us":630,"threads_started":10} I0108 16:43:14.532671 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 197594159 bytes on disk I0108 16:43:14.792264 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.259s user 0.000s sys 0.001s Metrics: {"cfile_init":5,"lbm_read_time_us":259294,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":6,"lbm_reads_lt_1ms":10} I0108 16:43:14.792402 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 57087952 bytes on disk I0108 16:43:14.906560 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.114s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":113966,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 16:43:14.906715 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.045038 I0108 16:43:16.486407 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 1.580s user 0.539s sys 0.021s Metrics: {"bytes_written":9024413,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":112,"dirs.run_wall_time_us":95898,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788800,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":9,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":907,"reinsert_count":0,"rows_written":513378,"thread_start_us":37,"threads_started":1,"update_count":19} I0108 16:43:16.486544 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 41964741 bytes of WAL I0108 16:43:16.486619 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 5 log segments from log reader I0108 16:43:16.486636 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000021 (ops 247933-260353) I0108 16:43:16.486667 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000022 (ops 260354-272721) I0108 16:43:16.486704 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000023 (ops 272722-285105) I0108 16:43:16.486724 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000024 (ops 285106-297525) I0108 16:43:16.486752 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000025 (ops 297526-309914) I0108 16:43:16.496588 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.010s user 0.000s sys 0.009s Metrics: {} I0108 16:43:16.496681 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 10250445 bytes on disk I0108 16:43:16.496793 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:43:16.496835 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.007407 I0108 16:43:20.624426 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 13.007s user 2.531s sys 0.224s Metrics: {"bytes_written":47154421,"cfile_cache_hit":107,"cfile_cache_hit_bytes":13552646,"cfile_cache_miss":6221,"cfile_cache_miss_bytes":179496452,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":532,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":90714,"drs_written":2,"lbm_read_time_us":7041987,"lbm_reads_1-10_ms":55,"lbm_reads_10-100_ms":161,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":6049,"lbm_write_time_us":2529556,"lbm_writes_1-10_ms":22,"lbm_writes_10-100_ms":53,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":4678,"num_input_rowsets":3,"rows_written":2680525,"thread_start_us":690,"threads_started":10} I0108 16:43:20.624657 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 57107314 bytes on disk I0108 16:43:20.694147 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.069s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":69304,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 16:43:24.947787 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033334 I0108 16:43:28.723728 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 27.284s user 2.637s sys 0.211s Metrics: {"bytes_written":47151136,"cfile_cache_hit":102,"cfile_cache_hit_bytes":12663639,"cfile_cache_miss":6230,"cfile_cache_miss_bytes":179765497,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":237,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1719715,"drs_written":2,"lbm_read_time_us":10159576,"lbm_reads_1-10_ms":32,"lbm_reads_10-100_ms":153,"lbm_reads_gt_100_ms":16,"lbm_reads_lt_1ms":6081,"lbm_write_time_us":12965675,"lbm_writes_1-10_ms":142,"lbm_writes_10-100_ms":237,"lbm_writes_gt_100_ms":19,"lbm_writes_lt_1ms":4358,"num_input_rowsets":3,"rows_written":2680448,"thread_start_us":464,"threads_started":10} I0108 16:43:28.723948 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 57104284 bytes on disk I0108 16:43:28.978382 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.254s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":254190,"lbm_reads_10-100_ms":4,"lbm_reads_lt_1ms":4} I0108 16:43:28.978480 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.049005 I0108 16:43:30.649718 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 5.702s user 1.516s sys 0.067s Metrics: {"bytes_written":25244121,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":106240,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3818149,"lbm_writes_1-10_ms":8,"lbm_writes_10-100_ms":54,"lbm_writes_gt_100_ms":9,"lbm_writes_lt_1ms":2470,"reinsert_count":0,"rows_written":1430547,"thread_start_us":32,"threads_started":1,"update_count":68} I0108 16:43:30.649860 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 59131239 bytes of WAL I0108 16:43:30.649952 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 7 log segments from log reader I0108 16:43:30.649966 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000048 (ops 278136-284034) I0108 16:43:30.650024 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000049 (ops 284035-289930) I0108 16:43:30.650091 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000050 (ops 289931-295906) I0108 16:43:30.650120 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000051 (ops 295907-301838) I0108 16:43:30.650138 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000052 (ops 301839-307912) I0108 16:43:30.650156 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000053 (ops 307913-313784) I0108 16:43:30.650203 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000054 (ops 313785-319691) I0108 16:43:30.663342 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.013s user 0.000s sys 0.013s Metrics: {} I0108 16:43:30.663480 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 28427118 bytes on disk I0108 16:43:30.663592 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":20,"lbm_reads_lt_1ms":4} I0108 16:43:30.663650 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.788911 I0108 16:43:33.094768 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 4.116s user 0.561s sys 0.032s Metrics: {"bytes_written":9051802,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":122,"dirs.run_wall_time_us":39280,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3384666,"lbm_writes_1-10_ms":39,"lbm_writes_10-100_ms":61,"lbm_writes_gt_100_ms":5,"lbm_writes_lt_1ms":824,"reinsert_count":0,"rows_written":514977,"thread_start_us":33,"threads_started":1,"update_count":8} I0108 16:43:33.094926 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 42025127 bytes of WAL I0108 16:43:33.094997 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 5 log segments from log reader I0108 16:43:33.095014 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000021 (ops 248030-260461) I0108 16:43:33.095060 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000022 (ops 260462-272798) I0108 16:43:33.095089 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000023 (ops 272799-285206) I0108 16:43:33.095109 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000024 (ops 285207-297635) I0108 16:43:33.095127 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000025 (ops 297636-310090) I0108 16:43:33.105547 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.011s user 0.000s sys 0.010s Metrics: {} I0108 16:43:33.105674 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 10274764 bytes on disk I0108 16:43:33.105792 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":16,"lbm_reads_lt_1ms":4} I0108 16:43:33.105841 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.007404 I0108 16:43:38.924708 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 22.428s user 2.595s sys 0.199s Metrics: {"bytes_written":47159886,"cfile_cache_hit":90,"cfile_cache_hit_bytes":10845941,"cfile_cache_miss":6242,"cfile_cache_miss_bytes":182213131,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":371,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1607987,"drs_written":2,"lbm_read_time_us":8224716,"lbm_reads_1-10_ms":35,"lbm_reads_10-100_ms":172,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":6073,"lbm_write_time_us":11002742,"lbm_writes_1-10_ms":40,"lbm_writes_10-100_ms":208,"lbm_writes_gt_100_ms":8,"lbm_writes_lt_1ms":4501,"num_input_rowsets":3,"rows_written":2680632,"spinlock_wait_cycles":1792,"thread_start_us":669,"threads_started":10} I0108 16:43:38.924916 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.036077 W0108 16:43:46.116338 24689 tablet_replica.cc:929] Time spent Committing in-flights took a long time.: real 0.728s user 0.001s sys 0.000s I0108 16:43:46.117209 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 7.192s user 1.611s sys 0.058s Metrics: {"bytes_written":24584300,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":121,"dirs.run_cpu_time_us":40,"dirs.run_wall_time_us":40,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4352208,"lbm_writes_1-10_ms":34,"lbm_writes_10-100_ms":75,"lbm_writes_gt_100_ms":7,"lbm_writes_lt_1ms":2359,"reinsert_count":0,"rows_written":1392818,"thread_start_us":91,"threads_started":1,"update_count":66} I0108 16:43:46.117347 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.789768 I0108 16:43:57.068233 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 23.962s user 2.638s sys 0.188s Metrics: {"bytes_written":47160277,"cfile_cache_hit":104,"cfile_cache_hit_bytes":12941086,"cfile_cache_miss":6228,"cfile_cache_miss_bytes":180122772,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":546,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":96258,"drs_written":2,"lbm_read_time_us":8273386,"lbm_reads_1-10_ms":40,"lbm_reads_10-100_ms":162,"lbm_reads_gt_100_ms":11,"lbm_reads_lt_1ms":6063,"lbm_write_time_us":12150145,"lbm_writes_1-10_ms":88,"lbm_writes_10-100_ms":211,"lbm_writes_gt_100_ms":19,"lbm_writes_lt_1ms":4440,"num_input_rowsets":3,"rows_written":2680874,"thread_start_us":803,"threads_started":10} I0108 16:43:57.068357 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 57102182 bytes on disk I0108 16:43:57.068481 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 16:43:57.068529 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.041017 I0108 16:44:04.919328 28561 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=953514, attempt_no=0}) took 1152ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":192,"cfile_cache_hit":26,"cfile_cache_hit_bytes":162937,"cfile_cache_miss":12,"cfile_cache_miss_bytes":49272,"lbm_read_time_us":1149025,"lbm_reads_10-100_ms":7,"lbm_reads_gt_100_ms":5,"num_ops":33,"prepare.queue_time_us":714,"prepare.run_cpu_time_us":225,"prepare.run_wall_time_us":345,"raft.queue_time_us":47,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":190,"replication_time_us":1821,"spinlock_wait_cycles":347392,"thread_start_us":255,"threads_started":3}]]} I0108 16:44:05.879971 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 8.811s user 1.628s sys 0.058s Metrics: {"bytes_written":24524901,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":66446,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6291885,"lbm_writes_1-10_ms":20,"lbm_writes_10-100_ms":110,"lbm_writes_gt_100_ms":11,"lbm_writes_lt_1ms":2329,"reinsert_count":0,"rows_written":1389398,"thread_start_us":67,"threads_started":1,"update_count":124} I0108 16:44:05.880110 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 58998946 bytes of WAL I0108 16:44:05.880175 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 7 log segments from log reader I0108 16:44:05.880192 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000048 (ops 277900-283739) I0108 16:44:05.880246 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000049 (ops 283740-289640) I0108 16:44:05.880304 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000050 (ops 289641-295514) I0108 16:44:05.880332 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000051 (ops 295515-301441) I0108 16:44:05.880352 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000052 (ops 301442-307416) I0108 16:44:05.880369 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000053 (ops 307417-313391) I0108 16:44:05.880390 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000054 (ops 313392-319281) I0108 16:44:05.892779 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.013s user 0.000s sys 0.013s Metrics: {} I0108 16:44:05.892861 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.789878 I0108 16:44:16.709853 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 46.046s user 8.319s sys 0.623s Metrics: {"bytes_written":154584012,"cfile_cache_hit":37,"cfile_cache_hit_bytes":373340,"cfile_cache_miss":20673,"cfile_cache_miss_bytes":635356842,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":1166,"dirs.run_cpu_time_us":828,"dirs.run_wall_time_us":361949,"drs_written":5,"lbm_read_time_us":28700845,"lbm_reads_1-10_ms":160,"lbm_reads_10-100_ms":586,"lbm_reads_gt_100_ms":37,"lbm_reads_lt_1ms":19974,"lbm_write_time_us":6067677,"lbm_writes_1-10_ms":42,"lbm_writes_10-100_ms":107,"lbm_writes_gt_100_ms":9,"lbm_writes_lt_1ms":15412,"num_input_rowsets":5,"rows_written":8787058,"spinlock_wait_cycles":1792,"thread_start_us":995,"threads_started":13} I0108 16:44:16.710048 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 187201309 bytes on disk I0108 16:44:16.897233 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.187s user 0.000s sys 0.001s Metrics: {"cfile_init":5,"lbm_read_time_us":186878,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":12} I0108 16:44:16.897344 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.283574 I0108 16:44:45.701675 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 59.584s user 8.405s sys 0.643s Metrics: {"bytes_written":154601140,"cfile_cache_hit":41,"cfile_cache_hit_bytes":969935,"cfile_cache_miss":20671,"cfile_cache_miss_bytes":634995029,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":830,"dirs.run_cpu_time_us":619,"dirs.run_wall_time_us":390733,"drs_written":5,"lbm_read_time_us":32624280,"lbm_reads_1-10_ms":150,"lbm_reads_10-100_ms":601,"lbm_reads_gt_100_ms":58,"lbm_reads_lt_1ms":19950,"lbm_write_time_us":15442989,"lbm_writes_1-10_ms":97,"lbm_writes_10-100_ms":299,"lbm_writes_gt_100_ms":17,"lbm_writes_lt_1ms":15158,"num_input_rowsets":5,"rows_written":8788091,"spinlock_wait_cycles":3724672,"thread_start_us":817,"threads_started":13} I0108 16:44:45.701866 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 58901508 bytes of WAL I0108 16:44:45.701933 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 7 log segments from log reader I0108 16:44:45.701969 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000048 (ops 279065-284997) I0108 16:44:45.702033 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000049 (ops 284998-290941) I0108 16:44:45.702109 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000050 (ops 290942-296840) I0108 16:44:45.702169 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000051 (ops 296841-302776) I0108 16:44:45.702221 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000052 (ops 302777-308783) I0108 16:44:45.702272 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000053 (ops 308784-314744) I0108 16:44:45.702322 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000054 (ops 314745-320682) I0108 16:44:45.723026 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.021s user 0.000s sys 0.018s Metrics: {} I0108 16:44:45.723220 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.283139 W0108 16:44:46.780815 28949 lock_manager.cc:368] Waited 1 seconds to obtain row lock on key \xa4\xa2\x17\x85\x0a.\x00\x00 cur holder: 0x597f500 I0108 16:44:48.505326 28998 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.131:58284 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=954799, attempt_no=0}) took 1582ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1549295,"cfile_cache_hit":10,"cfile_cache_hit_bytes":54758,"cfile_cache_miss":1,"cfile_cache_miss_bytes":4106,"lbm_read_time_us":32429,"lbm_reads_10-100_ms":1,"num_ops":9,"prepare.queue_time_us":30,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":70,"raft.queue_time_us":32,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":53,"replication_time_us":527,"spinlock_wait_cycles":8576}]]} I0108 16:44:53.352229 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 36.455s user 3.783s sys 0.336s Metrics: {"bytes_written":68002573,"cfile_cache_hit":19,"cfile_cache_hit_bytes":851719,"cfile_cache_miss":9093,"cfile_cache_miss_bytes":280201727,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":448,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":69834,"drs_written":2,"lbm_read_time_us":14937375,"lbm_reads_1-10_ms":87,"lbm_reads_10-100_ms":271,"lbm_reads_gt_100_ms":26,"lbm_reads_lt_1ms":8741,"lbm_write_time_us":16138735,"lbm_writes_1-10_ms":75,"lbm_writes_10-100_ms":280,"lbm_writes_gt_100_ms":25,"lbm_writes_lt_1ms":6465,"num_input_rowsets":2,"rows_written":3866645,"spinlock_wait_cycles":1906688,"thread_start_us":625,"threads_started":10} I0108 16:44:53.352443 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 82570312 bytes on disk I0108 16:44:53.596068 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.244s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":243439,"lbm_reads_10-100_ms":1,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":6} I0108 16:44:53.596228 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.034057 I0108 16:44:54.085268 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.489s user 0.009s sys 0.000s Metrics: {"bytes_written":139529,"cfile_init":1,"dirs.queue_time_us":68,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":479549,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":3,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":30,"rows_written":7285,"thread_start_us":31,"threads_started":1} I0108 16:44:54.085419 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 130940 bytes on disk I0108 16:44:54.085508 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4} I0108 16:44:54.085686 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.007027 I0108 16:45:09.411444 29127 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=971951, attempt_no=0}) took 1030ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":156,"cfile_cache_hit":146347,"cfile_cache_hit_bytes":1471759142,"num_ops":16979,"prepare.queue_time_us":12,"prepare.run_cpu_time_us":18208,"prepare.run_wall_time_us":63220,"raft.queue_time_us":31,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":169,"replication_time_us":8564,"spinlock_wait_cycles":2662272}]]} I0108 16:45:11.469926 29108 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=974942, attempt_no=0}) took 1049ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":9,"cfile_cache_hit":140713,"cfile_cache_hit_bytes":1414169626,"num_ops":16317,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":17428,"prepare.run_wall_time_us":45039,"raft.queue_time_us":8030,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":145,"replication_time_us":5934,"spinlock_wait_cycles":25656320}]]} I0108 16:45:12.117157 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 18.031s user 3.299s sys 0.401s Metrics: {"bytes_written":47290892,"cfile_cache_hit":31,"cfile_cache_hit_bytes":1925630,"cfile_cache_miss":6317,"cfile_cache_miss_bytes":194122274,"cfile_init":11,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":4503,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":197681,"drs_written":2,"lbm_read_time_us":3990465,"lbm_reads_1-10_ms":142,"lbm_reads_10-100_ms":84,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":6134,"lbm_write_time_us":1747481,"lbm_writes_1-10_ms":98,"lbm_writes_10-100_ms":35,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":4645,"num_input_rowsets":3,"reinsert_count":0,"rows_written":2687497,"spinlock_wait_cycles":57159168,"thread_start_us":389,"threads_started":10,"update_count":1021} I0108 16:45:12.117370 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.034586 I0108 16:45:12.359256 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.242s user 0.090s sys 0.010s Metrics: {"bytes_written":1185616,"cfile_init":1,"dirs.queue_time_us":21,"dirs.run_cpu_time_us":48,"dirs.run_wall_time_us":49,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":120783,"lbm_writes_1-10_ms":12,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":120,"rows_written":65880} I0108 16:45:12.359462 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.006541 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 330000 Mean: 2232.89 Percentiles: 0% (min) = 728 25% = 1355 50% (med) = 1522 75% = 1738 95% = 2384 99% = 8864 99.9% = 111680 99.99% = 696320 100% (max) = 1587501 *NOTE: some values were greater than highest trackable value I0108 16:45:13.370851 24342 linked_list-test-util.h:519] Written 16500000 rows in chain I0108 16:45:16.038877 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 30.316s user 4.702s sys 0.506s Metrics: {"bytes_written":68004197,"cfile_cache_hit":35,"cfile_cache_hit_bytes":3254726,"cfile_cache_miss":9077,"cfile_cache_miss_bytes":277841128,"cfile_init":9,"delta_iterators_relevant":4,"dirs.queue_time_us":130,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":117449,"drs_written":2,"lbm_read_time_us":8494411,"lbm_reads_1-10_ms":218,"lbm_reads_10-100_ms":119,"lbm_reads_gt_100_ms":22,"lbm_reads_lt_1ms":8754,"lbm_write_time_us":5302129,"lbm_writes_1-10_ms":160,"lbm_writes_10-100_ms":112,"lbm_writes_gt_100_ms":4,"lbm_writes_lt_1ms":6572,"num_input_rowsets":2,"rows_written":3866652,"spinlock_wait_cycles":74240256,"thread_start_us":73,"threads_started":2} I0108 16:45:16.039559 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.038292 I0108 16:45:16.289716 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 70.397s user 9.751s sys 0.872s Metrics: {"bytes_written":154630929,"cfile_cache_hit":98,"cfile_cache_hit_bytes":9501666,"cfile_cache_miss":20612,"cfile_cache_miss_bytes":626574836,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":14700,"dirs.run_cpu_time_us":547,"dirs.run_wall_time_us":244076,"drs_written":5,"lbm_read_time_us":24382711,"lbm_reads_1-10_ms":339,"lbm_reads_10-100_ms":451,"lbm_reads_gt_100_ms":33,"lbm_reads_lt_1ms":19877,"lbm_write_time_us":19182974,"lbm_writes_1-10_ms":218,"lbm_writes_10-100_ms":337,"lbm_writes_gt_100_ms":31,"lbm_writes_lt_1ms":14988,"num_input_rowsets":5,"rows_written":8789502,"spinlock_wait_cycles":349575040,"thread_start_us":577,"threads_started":13} I0108 16:45:16.289898 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 187274608 bytes on disk I0108 16:45:16.290056 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":30,"lbm_reads_lt_1ms":20} I0108 16:45:16.290138 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.283051 I0108 16:45:16.542047 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.502s user 0.095s sys 0.003s Metrics: {"bytes_written":1448250,"cfile_init":1,"dirs.queue_time_us":33,"dirs.run_cpu_time_us":50,"dirs.run_wall_time_us":50,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":355300,"lbm_writes_1-10_ms":11,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":140,"rows_written":80540} I0108 16:45:16.542157 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 8397839 bytes of WAL I0108 16:45:16.542227 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 1 log segments from log reader I0108 16:45:16.542246 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000026 (ops 312022-324478) I0108 16:45:16.544420 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.002s user 0.000s sys 0.003s Metrics: {} I0108 16:45:16.544498 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 228957912 bytes on disk I0108 16:45:16.912986 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.368s user 0.000s sys 0.000s Metrics: {"cfile_init":6,"lbm_read_time_us":368151,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":8,"lbm_reads_lt_1ms":12} I0108 16:45:16.913180 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.006386 I0108 16:45:19.282704 29164 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=981987, attempt_no=0}) took 2726ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":295,"cfile_cache_hit":1037,"cfile_cache_hit_bytes":4321472,"cfile_cache_miss":3865,"cfile_cache_miss_bytes":16011064,"lbm_read_time_us":2610831,"lbm_reads_1-10_ms":59,"lbm_reads_10-100_ms":50,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":3753,"num_ops":9112,"prepare.queue_time_us":6106,"prepare.run_cpu_time_us":9061,"prepare.run_wall_time_us":15148,"raft.queue_time_us":133,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":231,"replication_time_us":1605,"spinlock_wait_cycles":1522560}]]} I0108 16:45:28.670604 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 16.311s user 3.458s sys 0.373s Metrics: {"bytes_written":48319614,"cfile_cache_hit":55,"cfile_cache_hit_bytes":5502611,"cfile_cache_miss":6425,"cfile_cache_miss_bytes":194715501,"cfile_init":12,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":384,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":48996,"drs_written":2,"lbm_read_time_us":3519105,"lbm_reads_1-10_ms":146,"lbm_reads_10-100_ms":93,"lbm_reads_lt_1ms":6234,"lbm_write_time_us":605999,"lbm_writes_1-10_ms":123,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":4751,"num_input_rowsets":3,"reinsert_count":0,"rows_written":2746405,"spinlock_wait_cycles":75515520,"thread_start_us":612,"threads_started":10,"update_count":1052} I0108 16:45:28.670922 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 58600976 bytes on disk I0108 16:45:28.671236 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 16:45:28.671326 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 57250157 bytes on disk I0108 16:45:28.925526 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.254s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":254039,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":3,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3} I0108 16:45:30.677034 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033341 I0108 16:45:34.333094 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 3.656s user 0.771s sys 0.060s Metrics: {"bytes_written":8996401,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":22468,"dirs.run_cpu_time_us":99,"dirs.run_wall_time_us":20457,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":125230,"lbm_writes_1-10_ms":27,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":874,"reinsert_count":0,"rows_written":500181,"thread_start_us":30,"threads_started":1,"update_count":14523} I0108 16:45:34.333366 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 9711335 bytes on disk I0108 16:45:34.333743 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4} I0108 16:45:34.333887 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.722393 I0108 16:45:34.562237 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 17.649s user 3.496s sys 0.372s Metrics: {"bytes_written":48577078,"cfile_cache_hit":47,"cfile_cache_hit_bytes":4403161,"cfile_cache_miss":6463,"cfile_cache_miss_bytes":196792941,"cfile_init":12,"delete_count":0,"delta_iterators_relevant":6,"dirs.queue_time_us":1298,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":131352,"drs_written":2,"lbm_read_time_us":4285109,"lbm_reads_1-10_ms":174,"lbm_reads_10-100_ms":94,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":6239,"lbm_write_time_us":608237,"lbm_writes_1-10_ms":101,"lbm_writes_10-100_ms":4,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":4800,"num_input_rowsets":3,"reinsert_count":0,"rows_written":2760835,"spinlock_wait_cycles":60837120,"thread_start_us":824,"threads_started":10,"update_count":1817} I0108 16:45:34.562345 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.038354 I0108 16:45:35.548341 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.986s user 0.176s sys 0.019s Metrics: {"bytes_written":2528794,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":13336,"dirs.run_cpu_time_us":48,"dirs.run_wall_time_us":48,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685576,"lbm_writes_1-10_ms":14,"lbm_writes_10-100_ms":22,"lbm_writes_lt_1ms":235,"reinsert_count":0,"rows_written":140956,"thread_start_us":31,"threads_started":1,"update_count":3381} I0108 16:45:35.548548 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 8390865 bytes of WAL I0108 16:45:35.548666 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 1 log segments from log reader I0108 16:45:35.548977 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000026 (ops 309915-322385) I0108 16:45:35.551951 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.003s user 0.000s sys 0.002s Metrics: {} I0108 16:45:35.552048 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.005788 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 340000 Mean: 2237.19 Percentiles: 0% (min) = 728 25% = 1355 50% (med) = 1523 75% = 1738 95% = 2388 99% = 8968 99.9% = 114752 99.99% = 696320 100% (max) = 1587501 *NOTE: some values were greater than highest trackable value I0108 16:45:38.598255 24342 linked_list-test-util.h:519] Written 17000000 rows in chain I0108 16:45:43.360174 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 27.070s user 4.895s sys 0.588s Metrics: {"bytes_written":68008524,"cfile_cache_hit":60,"cfile_cache_hit_bytes":7041323,"cfile_cache_miss":9052,"cfile_cache_miss_bytes":274068029,"cfile_init":9,"delta_iterators_relevant":4,"dirs.queue_time_us":11573,"dirs.run_cpu_time_us":393,"dirs.run_wall_time_us":150623,"drs_written":3,"lbm_read_time_us":5348075,"lbm_reads_1-10_ms":257,"lbm_reads_10-100_ms":104,"lbm_reads_gt_100_ms":5,"lbm_reads_lt_1ms":8722,"lbm_write_time_us":2333886,"lbm_writes_1-10_ms":185,"lbm_writes_10-100_ms":43,"lbm_writes_lt_1ms":6646,"num_input_rowsets":2,"rows_written":3866737,"spinlock_wait_cycles":122497152,"thread_start_us":750,"threads_started":10} I0108 16:45:43.360661 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 82574853 bytes on disk I0108 16:45:43.365991 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":22,"lbm_reads_lt_1ms":12} I0108 16:45:43.366132 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.044983 I0108 16:45:43.993700 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.627s user 0.213s sys 0.009s Metrics: {"bytes_written":3074078,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":936,"dirs.run_cpu_time_us":121,"dirs.run_wall_time_us":42914,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":378582,"lbm_writes_1-10_ms":15,"lbm_writes_10-100_ms":12,"lbm_writes_lt_1ms":299,"reinsert_count":0,"rows_written":172120,"thread_start_us":31,"threads_started":1,"update_count":1507} I0108 16:45:43.993809 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 8392945 bytes of WAL I0108 16:45:43.993876 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 1 log segments from log reader I0108 16:45:43.993891 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000026 (ops 311317-323793) I0108 16:45:43.995859 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.002s user 0.000s sys 0.002s Metrics: {} I0108 16:45:43.995914 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.005473 I0108 16:45:44.966109 29261 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.23.197.130:36790 (ReqId={client: dbcb5a880e8742629e06e9264dfb379f, seq_no=1014709, attempt_no=0}) took 1045ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":129,"cfile_cache_hit":2969,"cfile_cache_hit_bytes":12395361,"cfile_cache_miss":1174,"cfile_cache_miss_bytes":4820444,"lbm_read_time_us":983481,"lbm_reads_1-10_ms":12,"lbm_reads_10-100_ms":6,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":1152,"num_ops":6697,"prepare.queue_time_us":232,"prepare.run_cpu_time_us":6326,"prepare.run_wall_time_us":17735,"raft.queue_time_us":97,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"replication_time_us":954}]]} I0108 16:45:54.915482 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 19.363s user 3.673s sys 0.375s Metrics: {"bytes_written":49616613,"cfile_cache_hit":69,"cfile_cache_hit_bytes":7729898,"cfile_cache_miss":6589,"cfile_cache_miss_bytes":197730386,"cfile_init":15,"delta_iterators_relevant":8,"dirs.queue_time_us":19225,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":98729,"drs_written":2,"lbm_read_time_us":4214671,"lbm_reads_1-10_ms":171,"lbm_reads_10-100_ms":95,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":6380,"lbm_write_time_us":707913,"lbm_writes_1-10_ms":138,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":4859,"num_input_rowsets":3,"rows_written":2821588,"spinlock_wait_cycles":62130176,"thread_start_us":611,"threads_started":10} I0108 16:45:54.915695 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.035777 I0108 16:45:59.581543 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 4.662s user 1.148s sys 0.073s Metrics: {"bytes_written":15109644,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":155,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":42745,"drs_written":1,"lbm_read_time_us":9,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1256879,"lbm_writes_1-10_ms":34,"lbm_writes_10-100_ms":21,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":1456,"reinsert_count":0,"rows_written":847544,"thread_start_us":120,"threads_started":1,"update_count":11085} I0108 16:45:59.583751 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 33679541 bytes of WAL I0108 16:45:59.584075 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 4 log segments from log reader I0108 16:45:59.584127 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000055 (ops 320683-326701) I0108 16:45:59.584888 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000056 (ops 326702-332599) I0108 16:45:59.585273 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000057 (ops 332600-338590) I0108 16:45:59.585336 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000058 (ops 338591-344513) I0108 16:45:59.665045 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.081s user 0.000s sys 0.013s Metrics: {} I0108 16:45:59.665128 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 60308486 bytes on disk I0108 16:45:59.717274 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.046s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":46362,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":4} I0108 16:45:59.717485 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.718814 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 350000 Mean: 2236.93 Percentiles: 0% (min) = 728 25% = 1356 50% (med) = 1524 75% = 1739 95% = 2394 99% = 9072 99.9% = 109568 I0108 16:46:02.241560 24342 linked_list-test-util.h:519] Written 17500000 rows in chain 99.99% = 686080 100% (max) = 1587501 *NOTE: some values were greater than highest trackable value I0108 16:46:04.890857 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 20.895s user 3.741s sys 0.455s Metrics: {"bytes_written":50153562,"cfile_cache_hit":66,"cfile_cache_hit_bytes":7313759,"cfile_cache_miss":6666,"cfile_cache_miss_bytes":200352605,"cfile_init":13,"delta_iterators_relevant":8,"dirs.queue_time_us":6185,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":108280,"drs_written":2,"lbm_read_time_us":4222980,"lbm_reads_1-10_ms":175,"lbm_reads_10-100_ms":70,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":6464,"lbm_write_time_us":694909,"lbm_writes_1-10_ms":159,"lbm_writes_10-100_ms":5,"lbm_writes_lt_1ms":4898,"num_input_rowsets":3,"rows_written":2852568,"spinlock_wait_cycles":163858816,"thread_start_us":147,"threads_started":2} I0108 16:46:04.891024 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 61015728 bytes on disk I0108 16:46:04.891254 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":15,"lbm_reads_lt_1ms":8} I0108 16:46:04.891337 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.042166 I0108 16:46:07.055030 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 2.163s user 0.332s sys 0.025s Metrics: {"bytes_written":4362755,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":27357,"dirs.run_cpu_time_us":47,"dirs.run_wall_time_us":47,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1012201,"lbm_writes_1-10_ms":26,"lbm_writes_10-100_ms":15,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":413,"reinsert_count":0,"rows_written":246750,"thread_start_us":30,"threads_started":1,"update_count":85} I0108 16:46:07.060150 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 16797910 bytes of WAL I0108 16:46:07.060252 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 2 log segments from log reader I0108 16:46:07.060263 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000026 (ops 310091-322541) I0108 16:46:07.060302 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000027 (ops 322542-334955) I0108 16:46:07.065594 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.005s user 0.000s sys 0.005s Metrics: {} I0108 16:46:07.070736 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 4887335 bytes on disk I0108 16:46:07.081140 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.010s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10253,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3} I0108 16:46:07.081223 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.004718 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 360000 Mean: 2232.89 Percentiles: 0% (min) = 728 25% = 1357 50% (med) = 1525 75% = 1741 95% = 2400 99% = 9128 99.9% = 109056 99.99% = 677888 100% (max) = 1587501 *NOTE: some values were greater than highest trackable value I0108 16:46:24.574805 24342 linked_list-test-util.h:519] Written 18000000 rows in chain I0108 16:46:28.235596 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 21.154s user 3.907s sys 0.425s Metrics: {"bytes_written":51460428,"cfile_cache_hit":97,"cfile_cache_hit_bytes":11931402,"cfile_cache_miss":6811,"cfile_cache_miss_bytes":200965028,"cfile_init":12,"delta_iterators_relevant":8,"dirs.queue_time_us":12142,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":52198,"drs_written":2,"lbm_read_time_us":2643801,"lbm_reads_1-10_ms":213,"lbm_reads_10-100_ms":41,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":6599,"lbm_write_time_us":735058,"lbm_writes_1-10_ms":169,"lbm_writes_10-100_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":5022,"num_input_rowsets":3,"rows_written":2927624,"spinlock_wait_cycles":169248384,"thread_start_us":1624,"threads_started":10} I0108 16:46:28.235978 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 62717215 bytes on disk I0108 16:46:28.237097 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 16:46:28.237193 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.039544 I0108 16:46:29.456717 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 55.123s user 10.680s sys 1.237s Metrics: {"bytes_written":144791794,"cfile_cache_hit":218,"cfile_cache_hit_bytes":27522732,"cfile_cache_miss":19192,"cfile_cache_miss_bytes":572539440,"cfile_init":21,"delta_iterators_relevant":12,"dirs.queue_time_us":12853,"dirs.run_cpu_time_us":618,"dirs.run_wall_time_us":201470,"drs_written":5,"lbm_read_time_us":8604977,"lbm_reads_1-10_ms":542,"lbm_reads_10-100_ms":178,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":18546,"lbm_write_time_us":1847694,"lbm_writes_1-10_ms":388,"lbm_writes_10-100_ms":11,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14196,"num_input_rowsets":5,"rows_written":8233580,"spinlock_wait_cycles":465505536,"thread_start_us":576,"threads_started":13} I0108 16:46:29.456845 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 16917260 bytes of WAL I0108 16:46:29.456884 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 2 log segments from log reader I0108 16:46:29.456892 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000055 (ops 319692-325725) I0108 16:46:29.456925 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000056 (ops 325726-331710) I0108 16:46:29.461874 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.005s user 0.000s sys 0.005s Metrics: {} I0108 16:46:29.461951 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 176080852 bytes on disk I0108 16:46:29.462085 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":35,"lbm_reads_lt_1ms":20} I0108 16:46:29.462131 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.309170 I0108 16:46:33.369623 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 5.132s user 1.775s sys 0.135s Metrics: {"bytes_written":23628413,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":43,"dirs.run_wall_time_us":44,"drs_written":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573052,"lbm_writes_1-10_ms":36,"lbm_writes_10-100_ms":14,"lbm_writes_lt_1ms":2325,"reinsert_count":0,"rows_written":1336294,"thread_start_us":126,"threads_started":1,"update_count":12266} I0108 16:46:33.369879 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 58967580 bytes of WAL I0108 16:46:33.369971 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 7 log segments from log reader I0108 16:46:33.370020 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000055 (ops 319282-325303) I0108 16:46:33.370098 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000056 (ops 325304-331236) I0108 16:46:33.370191 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000057 (ops 331237-337073) I0108 16:46:33.370250 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000058 (ops 337074-342965) I0108 16:46:33.370301 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000059 (ops 342966-348901) I0108 16:46:33.370353 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000060 (ops 348902-354803) I0108 16:46:33.370424 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000061 (ops 354804-360734) I0108 16:46:33.392391 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.022s user 0.001s sys 0.017s Metrics: {} I0108 16:46:33.392582 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.727242 ------------------------------------------------------------ Histogram for latency of insert operations (microseconds) Note: each insert is a batch of 50 rows. ------------------------------------------------------------ Count: 370000 Mean: 2228.63 Percentiles: 0% (min) = 728 25% = 1358 50% (med) = 1526 75% = 1742 95% = 2404 99% = 9184 99.9% = 106560 99.99% = 673792 100% (max) = 1587501 *NOTE: some values were greater than highest trackable value I0108 16:46:46.652436 24342 linked_list-test-util.h:519] Written 18500000 rows in chain I0108 16:46:56.469578 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 27.007s user 4.643s sys 0.541s Metrics: {"bytes_written":61302952,"cfile_cache_hit":129,"cfile_cache_hit_bytes":16840234,"cfile_cache_miss":8089,"cfile_cache_miss_bytes":237519882,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":9462,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":92501,"drs_written":2,"lbm_read_time_us":4235015,"lbm_reads_1-10_ms":273,"lbm_reads_10-100_ms":66,"lbm_reads_gt_100_ms":10,"lbm_reads_lt_1ms":7784,"lbm_write_time_us":1028875,"lbm_writes_1-10_ms":209,"lbm_writes_10-100_ms":12,"lbm_writes_lt_1ms":5958,"num_input_rowsets":3,"rows_written":3487079,"spinlock_wait_cycles":131934976,"thread_start_us":61,"threads_started":2} I0108 16:46:56.469714 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033996 I0108 16:46:57.140650 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 57.423s user 11.227s sys 1.214s Metrics: {"bytes_written":150865806,"cfile_cache_hit":267,"cfile_cache_hit_bytes":35310587,"cfile_cache_miss":19957,"cfile_cache_miss_bytes":589200561,"cfile_init":22,"delta_iterators_relevant":12,"dirs.queue_time_us":9931,"dirs.run_cpu_time_us":650,"dirs.run_wall_time_us":237987,"drs_written":5,"lbm_read_time_us":7440241,"lbm_reads_1-10_ms":566,"lbm_reads_10-100_ms":128,"lbm_reads_gt_100_ms":15,"lbm_reads_lt_1ms":19336,"lbm_write_time_us":2039729,"lbm_writes_1-10_ms":477,"lbm_writes_10-100_ms":13,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":14714,"num_input_rowsets":5,"rows_written":8580859,"spinlock_wait_cycles":584507264,"thread_start_us":858,"threads_started":13} I0108 16:46:57.140861 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.319268 I0108 16:46:58.485394 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 2.015s user 0.514s sys 0.029s Metrics: {"bytes_written":7688619,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":133,"dirs.run_cpu_time_us":90,"dirs.run_wall_time_us":85840,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1255070,"lbm_writes_1-10_ms":20,"lbm_writes_10-100_ms":17,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":748,"reinsert_count":0,"rows_written":434593,"thread_start_us":101,"threads_started":1,"update_count":2395} I0108 16:46:58.485522 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 33568838 bytes of WAL I0108 16:46:58.485580 24439 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 4 log segments from log reader I0108 16:46:58.485594 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000026 (ops 309939-322388) I0108 16:46:58.485625 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000027 (ops 322389-334759) I0108 16:46:58.485648 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000028 (ops 334760-347169) I0108 16:46:58.485674 24439 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000029 (ops 347170-359594) I0108 16:46:58.493453 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.008s user 0.000s sys 0.008s Metrics: {} I0108 16:46:58.493556 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 74677549 bytes on disk I0108 16:46:58.569916 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.076s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":76168,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":4} I0108 16:46:58.570050 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 8624763 bytes on disk I0108 16:46:58.647076 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.077s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76856,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":3} I0108 16:46:58.647228 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.002755 I0108 16:47:00.375859 24342 linked_list-test-util.h:528] Sample at HT timestamp: 6465662240255496192 Inserted count: 18767550 I0108 16:47:00.375886 24342 linked_list-test-util.h:532] Finished inserting list. Added 18767550 in chain I0108 16:47:00.375891 24342 linked_list-test-util.h:533] Last entries inserted had keys: I0108 16:47:00.375895 24342 linked_list-test-util.h:535] 0: 6922152026958135296 I0108 16:47:00.375903 24342 linked_list-test-util.h:535] 1: -2534618459284307967 I0108 16:47:00.375905 24342 linked_list-test-util.h:535] 2: 6455073653206089730 I0108 16:47:00.375914 24342 linked_list-test-util.h:535] 3: -3001556095548063741 I0108 16:47:00.375918 24342 linked_list-test-util.h:535] 4: -5536315292320661500 I0108 16:47:00.375921 24342 linked_list-test-util.h:535] 5: 3453799032634736645 I0108 16:47:00.375924 24342 linked_list-test-util.h:535] 6: -6002971453607706618 I0108 16:47:00.375927 24342 linked_list-test-util.h:535] 7: 2986861396370980871 I0108 16:47:00.375931 24342 linked_list-test-util.h:535] 8: 452383674575093768 I0108 16:47:00.375933 24342 linked_list-test-util.h:535] 9: -9004527549155770359 I0108 16:47:00.375936 24342 linked_list-test-util.h:535] 10: -14553961688662006 I0108 16:47:00.375938 24342 linked_list-test-util.h:535] 11: 8975138150801735691 I0108 16:47:00.375941 24342 linked_list-test-util.h:535] 12: 6440801166494138380 I0108 16:47:00.375944 24342 linked_list-test-util.h:535] 13: -3016250794725015539 I0108 16:47:00.375948 24342 linked_list-test-util.h:535] 14: 5973863530230382606 I0108 16:47:00.375952 24342 linked_list-test-util.h:535] 15: -3482906956012060657 I0108 16:47:00.375955 24342 linked_list-test-util.h:535] 16: -6017525415296368624 I0108 16:47:00.375958 24342 linked_list-test-util.h:535] 17: 2972166697194029073 I0108 16:47:00.375962 24342 linked_list-test-util.h:535] 18: -6484603789048414190 I0108 16:47:00.375965 24342 linked_list-test-util.h:535] 19: 2505510535906983955 I0108 16:47:00.375968 24342 linked_list-test-util.h:535] 20: -29248660865613804 I0108 16:47:00.375972 24342 linked_list-test-util.h:535] 21: 8960865664089784341 I0108 16:47:00.375975 24342 linked_list-test-util.h:535] 22: -496186297129369578 I0108 16:47:00.375978 24342 linked_list-test-util.h:535] 23: 8493928027826028567 I0108 16:47:00.375982 24342 linked_list-test-util.h:535] 24: 5959168831053430808 I0108 16:47:00.375985 24342 linked_list-test-util.h:535] 25: -3497601655189012455 I0108 16:47:00.375988 24342 linked_list-test-util.h:535] 26: 5492231194789675034 I0108 16:47:00.375993 24342 linked_list-test-util.h:535] 27: 2957753472993787931 I0108 16:47:00.375995 24342 linked_list-test-util.h:535] 28: -6499157750737076196 I0108 16:47:00.375999 24342 linked_list-test-util.h:535] 29: 2490815836730032157 I0108 16:47:00.376003 24342 linked_list-test-util.h:535] 30: -6965813912024121314 I0108 16:47:00.376006 24342 linked_list-test-util.h:535] 31: 8946170964912832543 I0108 16:47:00.376010 24342 linked_list-test-util.h:535] 32: -510458783841320928 I0108 16:47:00.376013 24342 linked_list-test-util.h:535] 33: 8479233328649076769 I0108 16:47:00.376016 24342 linked_list-test-util.h:535] 34: -977537157593366494 I0108 16:47:00.376020 24342 linked_list-test-util.h:535] 35: -3512155616877674461 I0108 16:47:00.376024 24342 linked_list-test-util.h:535] 36: 5477817970589433892 I0108 16:47:00.376027 24342 linked_list-test-util.h:535] 37: -3979093253141430235 I0108 16:47:00.376030 24342 linked_list-test-util.h:535] 38: 5010880334325678118 I0108 16:47:00.376034 24342 linked_list-test-util.h:535] 39: 2476261875041370151 I0108 16:47:00.376037 24342 linked_list-test-util.h:535] 40: -6980508611201073112 I0108 16:47:00.376040 24342 linked_list-test-util.h:535] 41: 2009183501289324585 I0108 16:47:00.376044 24342 linked_list-test-util.h:535] 42: -7447446247464828886 I0108 16:47:00.376047 24342 linked_list-test-util.h:535] 43: 8464538629472124971 I0108 16:47:00.376051 24342 linked_list-test-util.h:535] 44: -992091119282028500 I0108 16:47:00.376055 24342 linked_list-test-util.h:535] 45: 7997882468185079853 I0108 16:47:00.376058 24342 linked_list-test-util.h:535] 46: -1459028755545784274 I0108 16:47:00.376061 24342 linked_list-test-util.h:535] 47: -3993506477341671377 I0108 16:47:00.376065 24342 linked_list-test-util.h:535] 48: 4996185635148726320 I0108 16:47:00.376068 24342 linked_list-test-util.h:535] 49: -4460444113605427151 I0108 16:47:00.487407 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530220348799 usec, L: 1 Expected Rows: 7989850 I0108 16:47:00.872522 24342 linked_list-test-util.h:639] Calling callback on tserver 02a5a9f6601045b6b339c43c526fc29e I0108 16:47:11.461673 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 12.814s user 3.523s sys 0.266s Metrics: {"bytes_written":54845257,"cfile_cache_hit":88,"cfile_cache_hit_bytes":10161960,"cfile_cache_miss":7288,"cfile_cache_miss_bytes":216478468,"cfile_init":14,"delta_iterators_relevant":12,"dirs.queue_time_us":433,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":220608,"drs_written":2,"lbm_read_time_us":7193115,"lbm_reads_1-10_ms":87,"lbm_reads_10-100_ms":194,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":7057,"lbm_write_time_us":406500,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":2,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":5527,"num_input_rowsets":3,"rows_written":3122090,"spinlock_wait_cycles":896,"thread_start_us":671,"threads_started":11,"wal-append.queue_time_us":67} I0108 16:47:11.461854 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 67095339 bytes on disk I0108 16:47:11.461971 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":18,"lbm_reads_lt_1ms":8} I0108 16:47:12.462893 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033362 I0108 16:47:13.223917 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.761s user 0.333s sys 0.022s Metrics: {"bytes_written":6725635,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":60169,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11862,"lbm_writes_lt_1ms":689,"rows_written":380975,"spinlock_wait_cycles":896,"thread_start_us":60,"threads_started":2,"wal-append.queue_time_us":51} I0108 16:47:13.224005 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 33670469 bytes of WAL I0108 16:47:13.224063 24439 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 4 log segments from log reader I0108 16:47:13.224079 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000026 (ops 311894-324379) I0108 16:47:13.224110 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000027 (ops 324380-336915) I0108 16:47:13.224130 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000028 (ops 336916-349425) I0108 16:47:13.224149 24439 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000029 (ops 349426-362077) I0108 16:47:13.556859 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.333s user 0.000s sys 0.008s Metrics: {} I0108 16:47:13.557026 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 7550279 bytes on disk I0108 16:47:13.557163 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:47:13.557216 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.002680 I0108 16:47:14.373323 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 17.232s user 4.285s sys 0.343s Metrics: {"bytes_written":67409431,"cfile_cache_hit":97,"cfile_cache_hit_bytes":12014601,"cfile_cache_miss":8941,"cfile_cache_miss_bytes":267602263,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":676,"dirs.run_cpu_time_us":385,"dirs.run_wall_time_us":403160,"drs_written":2,"lbm_read_time_us":9598538,"lbm_reads_1-10_ms":144,"lbm_reads_10-100_ms":218,"lbm_reads_gt_100_ms":14,"lbm_reads_lt_1ms":8613,"lbm_write_time_us":760458,"lbm_writes_1-10_ms":28,"lbm_writes_10-100_ms":17,"lbm_writes_lt_1ms":6748,"num_input_rowsets":3,"rows_written":3835511,"thread_start_us":504,"threads_started":11,"wal-append.queue_time_us":47} I0108 16:47:14.373546 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 248174460 bytes on disk I0108 16:47:14.605082 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.231s user 0.001s sys 0.000s Metrics: {"cfile_init":6,"lbm_read_time_us":231207,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":8,"lbm_reads_lt_1ms":12} I0108 16:47:14.605209 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 58922074 bytes on disk I0108 16:47:14.656033 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.051s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":50628,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":3} I0108 16:47:16.657838 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033365 I0108 16:47:17.209076 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.551s user 0.427s sys 0.016s Metrics: {"bytes_written":6483041,"cfile_init":1,"dirs.queue_time_us":79,"dirs.run_cpu_time_us":104,"dirs.run_wall_time_us":36143,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12851,"lbm_writes_lt_1ms":666,"rows_written":367090,"thread_start_us":139,"threads_started":2,"wal-append.queue_time_us":121} I0108 16:47:17.209225 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 25176164 bytes of WAL I0108 16:47:17.209303 24689 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 3 log segments from log reader I0108 16:47:17.209313 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000027 (ops 324479-336980) I0108 16:47:17.209342 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000028 (ops 336981-349503) I0108 16:47:17.209359 24689 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000029 (ops 349504-362038) I0108 16:47:17.217826 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.009s user 0.000s sys 0.009s Metrics: {} I0108 16:47:17.217984 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 7259225 bytes on disk I0108 16:47:17.218108 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":11,"lbm_reads_lt_1ms":4} I0108 16:47:17.218189 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.002671 I0108 16:47:18.652523 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 45.260s user 10.427s sys 0.884s Metrics: {"bytes_written":159367810,"cfile_cache_hit":12340,"cfile_cache_hit_bytes":339782906,"cfile_cache_miss":9056,"cfile_cache_miss_bytes":319014862,"cfile_init":25,"delta_iterators_relevant":14,"dirs.queue_time_us":391,"dirs.run_cpu_time_us":684,"dirs.run_wall_time_us":459775,"drs_written":5,"lbm_read_time_us":13351490,"lbm_reads_1-10_ms":302,"lbm_reads_10-100_ms":288,"lbm_reads_gt_100_ms":18,"lbm_reads_lt_1ms":8548,"lbm_write_time_us":2753599,"lbm_writes_1-10_ms":251,"lbm_writes_10-100_ms":46,"lbm_writes_gt_100_ms":2,"lbm_writes_lt_1ms":15768,"num_input_rowsets":6,"rows_written":9069663,"spinlock_wait_cycles":69572864,"thread_start_us":217,"threads_started":6,"wal-append.queue_time_us":69} I0108 16:47:18.652660 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.333117 I0108 16:47:25.570359 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 12.013s user 3.334s sys 0.287s Metrics: {"bytes_written":54938726,"cfile_cache_hit":85,"cfile_cache_hit_bytes":9861603,"cfile_cache_miss":7293,"cfile_cache_miss_bytes":217446669,"cfile_init":13,"delta_iterators_relevant":10,"dirs.queue_time_us":534,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":124978,"drs_written":2,"lbm_read_time_us":7014646,"lbm_reads_1-10_ms":100,"lbm_reads_10-100_ms":198,"lbm_reads_gt_100_ms":5,"lbm_reads_lt_1ms":7042,"lbm_write_time_us":155876,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":5542,"num_input_rowsets":3,"rows_written":3127380,"thread_start_us":659,"threads_started":11,"wal-append.queue_time_us":59} I0108 16:47:25.570590 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 67227261 bytes on disk I0108 16:47:25.981474 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.411s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":410698,"lbm_reads_10-100_ms":4,"lbm_reads_gt_100_ms":1,"lbm_reads_lt_1ms":3} I0108 16:47:29.922008 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 12.704s user 3.304s sys 0.297s Metrics: {"bytes_written":54944110,"cfile_cache_hit":103,"cfile_cache_hit_bytes":12524070,"cfile_cache_miss":7283,"cfile_cache_miss_bytes":214873982,"cfile_init":13,"delta_iterators_relevant":10,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":140861,"drs_written":2,"lbm_read_time_us":7827570,"lbm_reads_1-10_ms":100,"lbm_reads_10-100_ms":191,"lbm_reads_gt_100_ms":14,"lbm_reads_lt_1ms":7030,"lbm_write_time_us":179002,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":5543,"num_input_rowsets":3,"rows_written":3127925,"thread_start_us":453,"threads_started":11,"wal-append.queue_time_us":64} I0108 16:47:29.922189 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 67230063 bytes on disk I0108 16:47:29.922312 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 16:47:33.434525 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 14.782s user 4.115s sys 0.256s Metrics: {"bytes_written":75997663,"cfile_cache_hit":6441,"cfile_cache_hit_bytes":182165107,"cfile_cache_miss":3753,"cfile_cache_miss_bytes":133006015,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":407,"dirs.run_cpu_time_us":334,"dirs.run_wall_time_us":230178,"drs_written":3,"lbm_read_time_us":4847837,"lbm_reads_1-10_ms":56,"lbm_reads_10-100_ms":130,"lbm_reads_gt_100_ms":5,"lbm_reads_lt_1ms":3606,"lbm_write_time_us":4772615,"lbm_writes_1-10_ms":53,"lbm_writes_10-100_ms":96,"lbm_writes_gt_100_ms":9,"lbm_writes_lt_1ms":7518,"num_input_rowsets":3,"rows_written":4325677,"spinlock_wait_cycles":768,"thread_start_us":703,"threads_started":12,"wal-append.queue_time_us":81} I0108 16:47:33.434733 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 259283015 bytes on disk I0108 16:47:33.599797 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.165s user 0.001s sys 0.000s Metrics: {"cfile_init":7,"lbm_read_time_us":164772,"lbm_reads_1-10_ms":9,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":14} I0108 16:47:34.488518 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033377 I0108 16:47:35.677047 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033369 I0108 16:47:36.056748 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 1.568s user 1.327s sys 0.068s Metrics: {"bytes_written":22833822,"cfile_init":1,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":59,"dirs.run_wall_time_us":59,"drs_written":1,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":41853,"lbm_writes_lt_1ms":2294,"rows_written":1291689,"thread_start_us":60,"threads_started":2,"wal-append.queue_time_us":55} I0108 16:47:36.056949 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 58751810 bytes of WAL I0108 16:47:36.057040 24439 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 7 log segments from log reader I0108 16:47:36.057088 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000057 (ops 331711-337558) I0108 16:47:36.057190 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000058 (ops 337559-343500) I0108 16:47:36.057324 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000059 (ops 343501-349367) I0108 16:47:36.057425 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000060 (ops 349368-355259) I0108 16:47:36.057490 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000061 (ops 355260-361210) I0108 16:47:36.057570 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000062 (ops 361211-366993) I0108 16:47:36.057653 24439 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000063 (ops 366994-372927) I0108 16:47:36.075870 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.019s user 0.001s sys 0.017s Metrics: {} I0108 16:47:36.076061 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.686263 I0108 16:47:36.261237 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.584s user 0.400s sys 0.019s Metrics: {"bytes_written":5409403,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":127,"dirs.run_wall_time_us":53276,"drs_written":1,"lbm_read_time_us":8,"lbm_reads_lt_1ms":4,"lbm_write_time_us":94223,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":551,"rows_written":305792,"thread_start_us":86,"threads_started":2,"wal-append.queue_time_us":52} I0108 16:47:36.261417 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 25160869 bytes of WAL I0108 16:47:36.261505 24689 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 3 log segments from log reader I0108 16:47:36.261555 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000027 (ops 322386-334774) I0108 16:47:36.261673 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000028 (ops 334775-347185) I0108 16:47:36.261752 24689 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000029 (ops 347186-359519) I0108 16:47:36.274104 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.013s user 0.003s sys 0.005s Metrics: {} I0108 16:47:36.274235 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.002923 I0108 16:47:43.725915 24342 linked_list-test-util.h:876] Done collecting results (7989850 rows in 42853.4ms) I0108 16:47:44.108778 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(97a6791627c941a48f3269f2d00d1eea): perf score=0.033365 I0108 16:47:44.619699 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.511s user 0.394s sys 0.009s Metrics: {"bytes_written":4867462,"cfile_init":1,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":113,"dirs.run_wall_time_us":49672,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":12557,"lbm_writes_lt_1ms":503,"rows_written":274812,"thread_start_us":94,"threads_started":2,"wal-append.queue_time_us":1956} I0108 16:47:44.619884 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(97a6791627c941a48f3269f2d00d1eea): free 25173185 bytes of WAL I0108 16:47:44.619966 24565 log_reader.cc:379] T 97a6791627c941a48f3269f2d00d1eea: removed 3 log segments from log reader I0108 16:47:44.620018 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000027 (ops 323794-336328) I0108 16:47:44.620121 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000028 (ops 336329-348798) I0108 16:47:44.620208 24565 log.cc:1062] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/97a6791627c941a48f3269f2d00d1eea/wal-000000029 (ops 348799-361328) I0108 16:47:44.628334 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.008s user 0.000s sys 0.008s Metrics: {} I0108 16:47:44.628516 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 5409289 bytes on disk I0108 16:47:44.628675 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":12,"lbm_reads_lt_1ms":4} I0108 16:47:44.628780 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea): perf score=1.003237 I0108 16:47:44.795934 24342 linked_list-test-util.h:667] Snapshot: P: 1578530220348799 usec, L: 1 verified. Result: OK I0108 16:47:44.796607 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530520348549 usec, L: 1 Expected Rows: 15130600 I0108 16:47:44.968524 24342 linked_list-test-util.h:639] Calling callback on tserver 02a5a9f6601045b6b339c43c526fc29e I0108 16:47:44.993815 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 8.719s user 3.318s sys 0.246s Metrics: {"bytes_written":54939014,"cfile_cache_hit":4869,"cfile_cache_hit_bytes":138727271,"cfile_cache_miss":2509,"cfile_cache_miss_bytes":88935813,"cfile_init":10,"delta_iterators_relevant":6,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":125192,"drs_written":2,"lbm_read_time_us":3198971,"lbm_reads_1-10_ms":76,"lbm_reads_10-100_ms":93,"lbm_reads_gt_100_ms":2,"lbm_reads_lt_1ms":2378,"lbm_write_time_us":1000343,"lbm_writes_1-10_ms":41,"lbm_writes_10-100_ms":28,"lbm_writes_lt_1ms":5475,"num_input_rowsets":3,"rows_written":3127380,"spinlock_wait_cycles":1536,"thread_start_us":748,"threads_started":11,"wal-append.queue_time_us":73} I0108 16:47:44.993995 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 67226300 bytes on disk I0108 16:47:44.994112 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":13,"lbm_reads_lt_1ms":8} I0108 16:47:57.338951 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 12.710s user 3.314s sys 0.269s Metrics: {"bytes_written":54938291,"cfile_cache_hit":17,"cfile_cache_hit_bytes":75634,"cfile_cache_miss":7363,"cfile_cache_miss_bytes":227731990,"cfile_init":11,"delta_iterators_relevant":6,"dirs.queue_time_us":725,"dirs.run_cpu_time_us":331,"dirs.run_wall_time_us":122128,"drs_written":2,"lbm_read_time_us":7097832,"lbm_reads_1-10_ms":97,"lbm_reads_10-100_ms":187,"lbm_reads_gt_100_ms":6,"lbm_reads_lt_1ms":7117,"lbm_write_time_us":1714679,"lbm_writes_1-10_ms":29,"lbm_writes_10-100_ms":44,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":5470,"num_input_rowsets":3,"rows_written":3127380,"spinlock_wait_cycles":1664,"thread_start_us":689,"threads_started":11,"wal-append.queue_time_us":65} I0108 16:47:57.339085 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 67226441 bytes on disk I0108 16:47:57.476398 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.137s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":137162,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":5} I0108 16:47:59.755962 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033382 I0108 16:48:01.108764 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 1.353s user 1.042s sys 0.052s Metrics: {"bytes_written":16728779,"cfile_init":1,"dirs.queue_time_us":8048,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":51813,"drs_written":1,"lbm_read_time_us":7,"lbm_reads_lt_1ms":4,"lbm_write_time_us":33734,"lbm_writes_lt_1ms":1678,"rows_written":943291,"thread_start_us":80,"threads_started":2,"wal-append.queue_time_us":69} I0108 16:48:01.108902 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 42074849 bytes of WAL I0108 16:48:01.108968 24689 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 5 log segments from log reader I0108 16:48:01.108981 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000059 (ops 344514-350528) I0108 16:48:01.109025 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000060 (ops 350529-356456) I0108 16:48:01.109056 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000061 (ops 356457-362430) I0108 16:48:01.109077 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000062 (ops 362431-368430) I0108 16:48:01.109109 24689 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000063 (ops 368431-374238) I0108 16:48:01.118362 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.009s user 0.000s sys 0.009s Metrics: {} I0108 16:48:01.118497 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 18624056 bytes on disk I0108 16:48:01.118614 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":10,"lbm_reads_lt_1ms":4} I0108 16:48:01.118729 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.665516 I0108 16:48:03.532533 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 27.456s user 9.375s sys 0.776s Metrics: {"bytes_written":158576940,"cfile_cache_hit":119,"cfile_cache_hit_bytes":12658696,"cfile_cache_miss":21151,"cfile_cache_miss_bytes":643927488,"cfile_init":21,"delta_iterators_relevant":10,"dirs.queue_time_us":736,"dirs.run_cpu_time_us":535,"dirs.run_wall_time_us":231468,"drs_written":5,"lbm_read_time_us":14506131,"lbm_reads_1-10_ms":281,"lbm_reads_10-100_ms":468,"lbm_reads_gt_100_ms":9,"lbm_reads_lt_1ms":20477,"lbm_write_time_us":358774,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":15988,"num_input_rowsets":5,"rows_written":9025166,"spinlock_wait_cycles":2304,"thread_start_us":635,"threads_started":14,"wal-append.queue_time_us":67} I0108 16:48:03.532747 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 193754827 bytes on disk I0108 16:48:03.532923 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":26,"lbm_reads_lt_1ms":20} I0108 16:48:03.532994 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.361810 I0108 16:48:07.234066 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033383 I0108 16:48:07.527209 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.293s user 0.180s sys 0.008s Metrics: {"bytes_written":3561817,"cfile_init":1,"dirs.queue_time_us":77,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":84592,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":367,"rows_written":200301,"spinlock_wait_cycles":768,"thread_start_us":66,"threads_started":2,"wal-append.queue_time_us":81} I0108 16:48:07.527308 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): free 16784839 bytes of WAL I0108 16:48:07.527361 24565 log_reader.cc:379] T c0c87b04bc0f4f37876458fbfb4bdf03: removed 2 log segments from log reader I0108 16:48:07.527369 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000028 (ops 334956-347364) I0108 16:48:07.527390 24565 log.cc:1062] T c0c87b04bc0f4f37876458fbfb4bdf03 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03/wal-000000029 (ops 347365-359733) I0108 16:48:07.531219 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.004s user 0.000s sys 0.004s Metrics: {} I0108 16:48:07.531316 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.003995 I0108 16:48:25.009817 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 21.477s user 4.960s sys 0.366s Metrics: {"bytes_written":83925997,"cfile_cache_hit":33,"cfile_cache_hit_bytes":1891559,"cfile_cache_miss":11231,"cfile_cache_miss_bytes":346766077,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":228,"dirs.run_cpu_time_us":404,"dirs.run_wall_time_us":174886,"drs_written":3,"lbm_read_time_us":14722700,"lbm_reads_1-10_ms":152,"lbm_reads_10-100_ms":341,"lbm_reads_gt_100_ms":16,"lbm_reads_lt_1ms":10770,"lbm_write_time_us":288335,"lbm_writes_1-10_ms":5,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":8464,"num_input_rowsets":3,"rows_written":4778730,"spinlock_wait_cycles":7168,"thread_start_us":378,"threads_started":12,"wal-append.queue_time_us":52} I0108 16:48:25.009968 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 102954215 bytes on disk I0108 16:48:25.010143 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":3,"lbm_read_time_us":25,"lbm_reads_lt_1ms":12} I0108 16:48:25.309312 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 17.778s user 3.534s sys 0.272s Metrics: {"bytes_written":54943427,"cfile_cache_hit":1187,"cfile_cache_hit_bytes":33721595,"cfile_cache_miss":6193,"cfile_cache_miss_bytes":194486597,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":189,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":52375,"drs_written":2,"lbm_read_time_us":8662469,"lbm_reads_1-10_ms":103,"lbm_reads_10-100_ms":185,"lbm_reads_gt_100_ms":8,"lbm_reads_lt_1ms":5929,"lbm_write_time_us":4342283,"lbm_writes_1-10_ms":29,"lbm_writes_10-100_ms":89,"lbm_writes_gt_100_ms":3,"lbm_writes_lt_1ms":5424,"num_input_rowsets":3,"rows_written":3127925,"spinlock_wait_cycles":384,"thread_start_us":584,"threads_started":11,"wal-append.queue_time_us":69} I0108 16:48:25.309540 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 67229146 bytes on disk I0108 16:48:25.309674 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":16,"lbm_reads_lt_1ms":8} I0108 16:48:30.860325 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 29.741s user 9.042s sys 0.778s Metrics: {"bytes_written":152464086,"cfile_cache_hit":43,"cfile_cache_hit_bytes":1687865,"cfile_cache_miss":20403,"cfile_cache_miss_bytes":630860879,"cfile_init":20,"delta_iterators_relevant":10,"dirs.queue_time_us":702,"dirs.run_cpu_time_us":676,"dirs.run_wall_time_us":340502,"drs_written":5,"lbm_read_time_us":17071934,"lbm_reads_1-10_ms":281,"lbm_reads_10-100_ms":435,"lbm_reads_gt_100_ms":18,"lbm_reads_lt_1ms":19749,"lbm_write_time_us":392915,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":15372,"num_input_rowsets":5,"rows_written":8676734,"spinlock_wait_cycles":1664,"thread_start_us":712,"threads_started":14,"wal-append.queue_time_us":85} I0108 16:48:30.860595 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 186371607 bytes on disk I0108 16:48:30.860790 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":5,"lbm_read_time_us":33,"lbm_reads_lt_1ms":20} I0108 16:48:30.860839 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.371082 I0108 16:48:33.566509 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.033388 I0108 16:48:34.244812 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: FlushMRSOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.678s user 0.531s sys 0.023s Metrics: {"bytes_written":8094006,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":129,"dirs.run_wall_time_us":47468,"drs_written":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4,"lbm_write_time_us":18597,"lbm_writes_lt_1ms":815,"rows_written":453113,"thread_start_us":62,"threads_started":2,"wal-append.queue_time_us":70} I0108 16:48:34.245038 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51): free 16734824 bytes of WAL I0108 16:48:34.245116 24565 log_reader.cc:379] T 2c3d9174c33442d28f28f4d0cda9fc51: removed 2 log segments from log reader I0108 16:48:34.245169 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000062 (ops 360735-366657) I0108 16:48:34.245242 24565 log.cc:1062] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e: Deleting log segment in path: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51/wal-000000063 (ops 366658-372542) I0108 16:48:34.250502 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: LogGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.005s user 0.000s sys 0.005s Metrics: {} I0108 16:48:34.250633 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 8872060 bytes on disk I0108 16:48:34.250958 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":13,"lbm_reads_lt_1ms":4} I0108 16:48:34.251047 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.685043 I0108 16:48:46.830437 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 15.969s user 4.638s sys 0.383s Metrics: {"bytes_written":83926277,"cfile_cache_hit":19,"cfile_cache_hit_bytes":116016,"cfile_cache_miss":11241,"cfile_cache_miss_bytes":348966062,"cfile_init":12,"delta_iterators_relevant":6,"dirs.queue_time_us":531,"dirs.run_cpu_time_us":311,"dirs.run_wall_time_us":151969,"drs_written":3,"lbm_read_time_us":9712556,"lbm_reads_1-10_ms":154,"lbm_reads_10-100_ms":278,"lbm_reads_gt_100_ms":4,"lbm_reads_lt_1ms":10853,"lbm_write_time_us":289363,"lbm_writes_1-10_ms":3,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":8466,"num_input_rowsets":3,"rows_written":4778790,"spinlock_wait_cycles":896,"thread_start_us":528,"threads_started":12,"wal-append.queue_time_us":83} I0108 16:48:46.830654 24795 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 102954330 bytes on disk I0108 16:48:46.913275 24689 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.083s user 0.000s sys 0.001s Metrics: {"cfile_init":3,"lbm_read_time_us":82392,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":6} I0108 16:48:52.153592 24342 linked_list-test-util.h:876] Done collecting results (15130600 rows in 67185.1ms) I0108 16:48:54.069635 24342 linked_list-test-util.h:667] Snapshot: P: 1578530520348549 usec, L: 1 verified. Result: OK I0108 16:48:54.071666 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530820374877 usec, L: 1 Expected Rows: 18767550 I0108 16:48:54.130798 24342 linked_list-test-util.h:639] Calling callback on tserver e61264bf360a4a62a3a5e3b9f2875f5b I0108 16:48:54.556710 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 20.306s user 8.284s sys 0.595s Metrics: {"bytes_written":151905255,"cfile_cache_hit":9946,"cfile_cache_hit_bytes":282852212,"cfile_cache_miss":10436,"cfile_cache_miss_bytes":349590306,"cfile_init":13,"delta_iterators_relevant":12,"dirs.queue_time_us":1085,"dirs.run_cpu_time_us":570,"dirs.run_wall_time_us":219847,"drs_written":5,"lbm_read_time_us":8501339,"lbm_reads_1-10_ms":136,"lbm_reads_10-100_ms":222,"lbm_reads_gt_100_ms":7,"lbm_reads_lt_1ms":10123,"lbm_write_time_us":861513,"lbm_writes_1-10_ms":17,"lbm_writes_10-100_ms":19,"lbm_writes_lt_1ms":15283,"num_input_rowsets":6,"rows_written":8645522,"spinlock_wait_cycles":2304,"thread_start_us":796,"threads_started":14,"wal-append.queue_time_us":73} I0108 16:48:54.556907 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51): perf score=0.467538 I0108 16:48:58.536545 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=0.033348 I0108 16:48:58.543593 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: FlushMRSOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.007s user 0.006s sys 0.000s Metrics: {"bytes_written":106706,"cfile_init":1,"dirs.queue_time_us":75,"drs_written":1,"lbm_read_time_us":4,"lbm_reads_lt_1ms":4,"lbm_write_time_us":342,"lbm_writes_lt_1ms":34,"rows_written":5835,"thread_start_us":57,"threads_started":2,"wal-append.queue_time_us":41} I0108 16:48:58.543690 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03): perf score=1.005816 I0108 16:49:03.602932 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: CompactRowSetsOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 5.059s user 2.730s sys 0.164s Metrics: {"bytes_written":54943845,"cfile_cache_hit":7165,"cfile_cache_hit_bytes":203985620,"cfile_cache_miss":221,"cfile_cache_miss_bytes":25225376,"cfile_init":8,"delta_iterators_relevant":6,"dirs.queue_time_us":615,"dirs.run_cpu_time_us":291,"dirs.run_wall_time_us":104055,"drs_written":2,"lbm_read_time_us":217005,"lbm_reads_1-10_ms":8,"lbm_reads_10-100_ms":7,"lbm_reads_lt_1ms":238,"lbm_write_time_us":1209952,"lbm_writes_1-10_ms":29,"lbm_writes_10-100_ms":43,"lbm_writes_lt_1ms":5473,"num_input_rowsets":3,"rows_written":3127925,"spinlock_wait_cycles":40832,"thread_start_us":662,"threads_started":10,"wal-append.queue_time_us":70} I0108 16:49:03.603108 24546 maintenance_manager.cc:326] P e61264bf360a4a62a3a5e3b9f2875f5b: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 67229405 bytes on disk I0108 16:49:03.700956 24439 maintenance_manager.cc:527] P e61264bf360a4a62a3a5e3b9f2875f5b: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.098s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":97702,"lbm_reads_1-10_ms":2,"lbm_reads_10-100_ms":3,"lbm_reads_lt_1ms":3} I0108 16:49:11.164757 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: CompactRowSetsOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 16.608s user 7.582s sys 0.546s Metrics: {"bytes_written":135829951,"cfile_cache_hit":3831,"cfile_cache_hit_bytes":107659217,"cfile_cache_miss":14393,"cfile_cache_miss_bytes":459156281,"cfile_init":15,"delta_iterators_relevant":8,"dirs.queue_time_us":1387,"dirs.run_cpu_time_us":619,"dirs.run_wall_time_us":187000,"drs_written":5,"lbm_read_time_us":6464925,"lbm_reads_1-10_ms":178,"lbm_reads_10-100_ms":209,"lbm_reads_gt_100_ms":3,"lbm_reads_lt_1ms":14063,"lbm_write_time_us":380439,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":3,"lbm_writes_lt_1ms":13718,"num_input_rowsets":4,"rows_written":7733504,"spinlock_wait_cycles":768,"thread_start_us":694,"threads_started":13,"wal-append.queue_time_us":62} I0108 16:49:11.164924 24671 maintenance_manager.cc:326] P 02a5a9f6601045b6b339c43c526fc29e: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 269564771 bytes on disk I0108 16:49:11.319185 24565 maintenance_manager.cc:527] P 02a5a9f6601045b6b339c43c526fc29e: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.154s user 0.001s sys 0.000s Metrics: {"cfile_init":8,"lbm_read_time_us":153902,"lbm_reads_1-10_ms":9,"lbm_reads_10-100_ms":5,"lbm_reads_lt_1ms":18} I0108 16:49:33.872174 24342 linked_list-test-util.h:876] Done collecting results (18767550 rows in 39741.4ms) I0108 16:49:36.248746 24342 linked_list-test-util.h:667] Snapshot: P: 1578530820374877 usec, L: 1 verified. Result: OK I0108 16:49:36.249542 24342 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 18767550 I0108 16:50:09.183171 24342 linked_list-test-util.h:876] Done collecting results (18767550 rows in 32550.9ms) I0108 16:50:11.513263 24342 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0108 16:50:11.513337 24342 linked_list-test-util.h:801] Successfully verified 18767550 rows I0108 16:50:11.517906 24342 linked_list-test.cc:194] Successfully verified 18767550 rows before killing any servers. I0108 16:50:11.519505 24342 external_mini_cluster.cc:721] 3 TS(s) registered with all masters I0108 16:50:11.519757 24342 ts_itest-base.cc:233] Waiting for 3 tablets on tserver e61264bf360a4a62a3a5e3b9f2875f5b to finish bootstrapping I0108 16:50:11.521258 24342 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 02a5a9f6601045b6b339c43c526fc29e to finish bootstrapping I0108 16:50:11.522665 24342 ts_itest-base.cc:233] Waiting for 3 tablets on tserver 72e6f176048945dea46e0f4305315d3c to finish bootstrapping I0108 16:50:11.524042 24342 linked_list-test.cc:199] Will restart the tablet server during verification scan. I0108 16:50:11.524416 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530220348799 usec, L: 1 Expected Rows: 7989850 I0108 16:50:11.593397 24342 linked_list-test-util.h:639] Calling callback on tserver 72e6f176048945dea46e0f4305315d3c I0108 16:50:11.593492 24342 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24673 W0108 16:50:11.643713 24432 connection.cc:655] client connection to 127.23.197.131:49342 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W0108 16:50:11.643748 24432 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342): Couldn't send request to peer 72e6f176048945dea46e0f4305315d3c. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I0108 16:50:11.644850 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.23.197.131:49342 --local_ip_for_outbound_sockets=127.23.197.131 --webserver_port=52477 --webserver_interface=127.23.197.131 --tserver_master_addrs=127.23.197.190:59202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} W0108 16:50:11.651211 24558 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P 02a5a9f6601045b6b339c43c526fc29e -> Peer 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342): Couldn't send request to peer 72e6f176048945dea46e0f4305315d3c. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.131:49342: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:50:11.656731 30224 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:50:11.656828 30224 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:50:11.657599 30224 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:50:11.657622 30224 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.197.131 I0108 16:50:11.658581 30224 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.197.131:49342 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/info.pb --webserver_interface=127.23.197.131 --webserver_port=52477 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.197.190:59202 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.30224 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.197.131 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:50:11.660693 30224 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:50:11.661832 30224 hybrid_clock.cc:186] HybridClock initialized: now 1578531011661812 us; error 26 us; skew 500 ppm I0108 16:50:11.661901 30224 fs_manager.cc:263] Metadata directory not provided I0108 16:50:11.661919 30224 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:50:11.664099 30224 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s I0108 16:50:11.664232 30224 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:50:11.664283 30224 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:50:11.666380 30224 fs_manager.cc:434] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s I0108 16:50:11.666579 30224 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal uuid: "72e6f176048945dea46e0f4305315d3c" format_stamp: "Formatted at 2020-01-09 00:32:00 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:50:11.666610 30224 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/data Total live blocks: 66 Total live bytes: 733678692 Total live bytes (after alignment): 733835264 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:50:11.696772 30224 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:50:11.696979 30224 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0108 16:50:11.697752 30224 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0108 16:50:11.697768 30224 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0108 16:50:11.698065 30247 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Bootstrap starting. I0108 16:50:11.698287 30224 ts_tablet_manager.cc:415] Registered 3 tablets I0108 16:50:11.702761 30224 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.131:49342 I0108 16:50:11.702955 30224 webserver.cc:382] Webserver started at http://127.23.197.131:52477/ using document root and password file I0108 16:50:11.703063 30224 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/data/info.pb I0108 16:50:11.705754 30352 heartbeater.cc:322] Connected to a master server at 127.23.197.190:59202 I0108 16:50:11.705771 30352 heartbeater.cc:406] Registering TS with master... I0108 16:50:11.705818 30352 heartbeater.cc:455] Master 127.23.197.190:59202 requested a full tablet report, sending... I0108 16:50:11.706012 24372 ts_manager.cc:192] Re-registered known tserver with Master: 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:50:11.706543 24372 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.197.131:50985 I0108 16:50:11.708204 24342 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 30224 I0108 16:50:11.712101 30269 tablet_service.cc:2571] Scan: Not found: Scanner 26f1632374e145478fc6b110203f31e0 not found (it may have expired): call sequence id=1, remote={username='jenkins'} at 127.23.197.131:58561 W0108 16:50:11.712255 24342 client.cc:1692] Attempting to retry Scanner { table: TestTable, tablet: 2c3d9174c33442d28f28f4d0cda9fc51, projection: ( rand_key INT64 NOT NULL, link_to INT64 NOT NULL, updated BOOL NOT NULL, PRIMARY KEY () ), scan_spec: } elsewhere. I0108 16:50:11.800961 30247 log.cc:815] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Log is configured to *not* fsync() on all Append() calls I0108 16:50:12.024185 30247 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6239 overwritten=0 applied=6239 ignored=6239} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0108 16:50:12.024273 30247 log.cc:1202] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/c0c87b04bc0f4f37876458fbfb4bdf03.recovery-1578531012024228 I0108 16:50:12.024289 30247 tablet_bootstrap.cc:439] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Bootstrap complete. I0108 16:50:12.024324 30247 ts_tablet_manager.cc:1105] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Time spent bootstrapping tablet: real 0.326s user 0.202s sys 0.067s I0108 16:50:12.025053 30247 raft_consensus.cc:341] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.025075 30247 raft_consensus.cc:712] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Initialized, Role: FOLLOWER I0108 16:50:12.025137 30247 consensus_queue.cc:229] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 368277, Last appended: 1.368277, Last appended by leader: 368277, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.025283 30247 ts_tablet_manager.cc:1133] T c0c87b04bc0f4f37876458fbfb4bdf03 P 72e6f176048945dea46e0f4305315d3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0108 16:50:12.025282 30352 heartbeater.cc:447] Master 127.23.197.190:59202 was elected leader, sending a full tablet report... I0108 16:50:12.025313 30247 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Bootstrap starting. I0108 16:50:12.203631 30353 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03): 67230063 bytes on disk I0108 16:50:12.251713 30246 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(c0c87b04bc0f4f37876458fbfb4bdf03) complete. Timing: real 0.048s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":47853,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":4} I0108 16:50:12.314548 30247 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8552 overwritten=0 applied=8552 ignored=8552} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0108 16:50:12.314632 30247 log.cc:1202] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/97a6791627c941a48f3269f2d00d1eea.recovery-1578531012314589 I0108 16:50:12.314641 30247 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Bootstrap complete. I0108 16:50:12.314669 30247 ts_tablet_manager.cc:1105] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Time spent bootstrapping tablet: real 0.289s user 0.231s sys 0.049s I0108 16:50:12.314759 30247 raft_consensus.cc:341] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.314774 30247 raft_consensus.cc:712] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Initialized, Role: FOLLOWER I0108 16:50:12.314816 30247 consensus_queue.cc:229] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 368071, Last appended: 1.368071, Last appended by leader: 368071, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.314883 30247 ts_tablet_manager.cc:1133] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I0108 16:50:12.314947 30247 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Bootstrap starting. I0108 16:50:12.502072 30353 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea): 67226300 bytes on disk I0108 16:50:12.538805 30247 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1974 overwritten=0 applied=1974 ignored=1974} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates I0108 16:50:12.538900 30247 log.cc:1202] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: --skip_remove_old_recovery_dir enabled. NOT deleting /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-2/wal/wals/2c3d9174c33442d28f28f4d0cda9fc51.recovery-1578531012538854 I0108 16:50:12.538914 30247 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Bootstrap complete. I0108 16:50:12.538947 30247 ts_tablet_manager.cc:1105] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Time spent bootstrapping tablet: real 0.224s user 0.129s sys 0.021s I0108 16:50:12.539038 30247 raft_consensus.cc:341] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.539057 30247 raft_consensus.cc:712] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Initialized, Role: FOLLOWER I0108 16:50:12.539099 30247 consensus_queue.cc:229] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 376212, Last appended: 1.376212, Last appended by leader: 376212, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.539201 30247 ts_tablet_manager.cc:1133] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I0108 16:50:12.543139 30246 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(97a6791627c941a48f3269f2d00d1eea) complete. Timing: real 0.041s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":40900,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":4} I0108 16:50:12.543244 30353 maintenance_manager.cc:326] P 72e6f176048945dea46e0f4305315d3c: Scheduling UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51): 269569007 bytes on disk I0108 16:50:12.601914 30246 maintenance_manager.cc:527] P 72e6f176048945dea46e0f4305315d3c: UndoDeltaBlockGCOp(2c3d9174c33442d28f28f4d0cda9fc51) complete. Timing: real 0.059s user 0.000s sys 0.001s Metrics: {"cfile_init":7,"lbm_read_time_us":58462,"lbm_reads_1-10_ms":3,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":23} I0108 16:50:12.813230 30362 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72e6f176048945dea46e0f4305315d3c) I0108 16:50:12.813308 30362 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.813441 30362 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:50:12.813597 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 2 candidate_status { last_received { term: 1 index: 368071 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" is_pre_election: true I0108 16:50:12.813634 30309 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 2 candidate_status { last_received { term: 1 index: 368071 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" is_pre_election: true I0108 16:50:12.813657 30309 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a5a9f6601045b6b339c43c526fc29e in term 1. I0108 16:50:12.813755 24558 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02a5a9f6601045b6b339c43c526fc29e, 72e6f176048945dea46e0f4305315d3c; no voters: e61264bf360a4a62a3a5e3b9f2875f5b I0108 16:50:12.813820 30362 raft_consensus.cc:2696] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Leader pre-election won for term 2 I0108 16:50:12.813838 30362 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 72e6f176048945dea46e0f4305315d3c) I0108 16:50:12.813845 30362 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 1 FOLLOWER]: Advancing to term 2 I0108 16:50:12.814416 30362 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.814474 30362 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 2 election: Requested vote from peers e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:50:12.814581 30309 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 2 candidate_status { last_received { term: 1 index: 368071 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" I0108 16:50:12.814597 30309 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Advancing to term 2 I0108 16:50:12.814599 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "02a5a9f6601045b6b339c43c526fc29e" candidate_term: 2 candidate_status { last_received { term: 1 index: 368071 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" I0108 16:50:12.815806 30309 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a5a9f6601045b6b339c43c526fc29e in term 2. I0108 16:50:12.815898 24558 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 02a5a9f6601045b6b339c43c526fc29e, 72e6f176048945dea46e0f4305315d3c; no voters: e61264bf360a4a62a3a5e3b9f2875f5b I0108 16:50:12.815948 30362 raft_consensus.cc:2696] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 2 FOLLOWER]: Leader election won for term 2 I0108 16:50:12.816002 30362 raft_consensus.cc:672] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [term 2 LEADER]: Becoming Leader. State: Replica: 02a5a9f6601045b6b339c43c526fc29e, State: Running, Role: LEADER I0108 16:50:12.816054 30362 consensus_queue.cc:206] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368071, Committed index: 368071, Last appended: 1.368071, Last appended by leader: 368071, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.816502 24372 catalog_manager.cc:4143] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e reported cstate change: term changed from 1 to 2, leader changed from 72e6f176048945dea46e0f4305315d3c (127.23.197.131) to 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130). New cstate: current_term: 2 leader_uuid: "02a5a9f6601045b6b339c43c526fc29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } health_report { overall_health: UNKNOWN } } } I0108 16:50:12.872066 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.303604 I0108 16:50:12.873219 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"bytes_written":438,"delete_count":0,"lbm_write_time_us":40,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1} I0108 16:50:12.902658 30190 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72e6f176048945dea46e0f4305315d3c) I0108 16:50:12.902721 30190 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:50:12.902815 30190 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:50:12.902942 30309 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 368071 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" is_pre_election: true I0108 16:50:12.902968 30309 raft_consensus.cc:2321] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e61264bf360a4a62a3a5e3b9f2875f5b in current term 2: Already voted for candidate 02a5a9f6601045b6b339c43c526fc29e in this term. I0108 16:50:12.903053 24627 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 368071 } } ignore_live_leader: false dest_uuid: "02a5a9f6601045b6b339c43c526fc29e" is_pre_election: true I0108 16:50:12.903188 24434 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e61264bf360a4a62a3a5e3b9f2875f5b; no voters: 02a5a9f6601045b6b339c43c526fc29e, 72e6f176048945dea46e0f4305315d3c I0108 16:50:12.903266 30190 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Advancing to term 2 I0108 16:50:12.904474 30190 raft_consensus.cc:2644] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0108 16:50:13.304879 30309 raft_consensus.cc:1195] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Refusing update from remote peer 02a5a9f6601045b6b339c43c526fc29e: Log matching property violated. Preceding OpId in replica: term: 1 index: 368071. Preceding OpId from leader: term: 2 index: 368072. (index mismatch) I0108 16:50:13.305101 30365 consensus_queue.cc:982] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368072, Last known committed idx: 368071, Time since last communication: 0.000s I0108 16:50:13.306216 24501 raft_consensus.cc:1195] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Refusing update from remote peer 02a5a9f6601045b6b339c43c526fc29e: Log matching property violated. Preceding OpId in replica: term: 1 index: 368071. Preceding OpId from leader: term: 2 index: 368072. (index mismatch) I0108 16:50:13.306393 30365 consensus_queue.cc:982] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368072, Last known committed idx: 368071, Time since last communication: 0.000s I0108 16:50:34.135072 24342 linked_list-test-util.h:876] Done collecting results (7989850 rows in 22541.7ms) I0108 16:50:35.078780 24342 linked_list-test-util.h:667] Snapshot: P: 1578530220348799 usec, L: 1 verified. Result: OK I0108 16:50:35.079442 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530520348549 usec, L: 1 Expected Rows: 15130600 I0108 16:50:35.080963 24342 linked_list-test-util.h:639] Calling callback on tserver 72e6f176048945dea46e0f4305315d3c I0108 16:51:10.366858 24342 linked_list-test-util.h:876] Done collecting results (15130600 rows in 35285.9ms) I0108 16:51:12.210757 24342 linked_list-test-util.h:667] Snapshot: P: 1578530520348549 usec, L: 1 verified. Result: OK I0108 16:51:12.211305 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530820374877 usec, L: 1 Expected Rows: 18767550 I0108 16:51:12.241457 24342 linked_list-test-util.h:639] Calling callback on tserver e61264bf360a4a62a3a5e3b9f2875f5b I0108 16:51:46.084367 24342 linked_list-test-util.h:876] Done collecting results (18767550 rows in 33842.9ms) I0108 16:51:48.461122 24342 linked_list-test-util.h:667] Snapshot: P: 1578530820374877 usec, L: 1 verified. Result: OK I0108 16:51:48.461660 24342 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 18767550 I0108 16:52:20.269071 24342 linked_list-test-util.h:876] Done collecting results (18767550 rows in 31805.2ms) I0108 16:52:22.604470 24342 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0108 16:52:22.604547 24342 linked_list-test-util.h:801] Successfully verified 18767550 rows I0108 16:52:22.604647 24342 linked_list-test.cc:204] Done with tserver restart test. I0108 16:52:22.609031 24342 linked_list-test.cc:210] Will kill the tablet server during verification scan. I0108 16:52:22.609460 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530220348799 usec, L: 1 Expected Rows: 7989850 I0108 16:52:22.610622 24342 linked_list-test-util.h:639] Calling callback on tserver 02a5a9f6601045b6b339c43c526fc29e I0108 16:52:22.610679 24342 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24549 I0108 16:52:22.661198 24342 meta_cache.cc:1038] Marking tablet server 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) as failed. W0108 16:52:22.661265 24342 meta_cache.cc:234] Tablet 2c3d9174c33442d28f28f4d0cda9fc51: Replica 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) W0108 16:52:22.661285 24342 client.cc:1692] Attempting to retry Scanner { table: TestTable, tablet: 2c3d9174c33442d28f28f4d0cda9fc51, projection: ( rand_key INT64 NOT NULL, link_to INT64 NOT NULL, updated BOOL NOT NULL, PRIMARY KEY () ), scan_spec: } elsewhere. W0108 16:52:22.685808 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0108 16:52:23.508525 30508 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02a5a9f6601045b6b339c43c526fc29e) I0108 16:52:23.508587 30508 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.508687 30508 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) W0108 16:52:23.508857 24434 leader_election.cc:339] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) I0108 16:52:23.508882 30309 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 3 candidate_status { last_received { term: 2 index: 368072 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" is_pre_election: true I0108 16:52:23.509001 24432 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e61264bf360a4a62a3a5e3b9f2875f5b; no voters: 02a5a9f6601045b6b339c43c526fc29e, 72e6f176048945dea46e0f4305315d3c I0108 16:52:23.509066 30508 raft_consensus.cc:2644] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0108 16:52:23.755712 30508 raft_consensus.cc:477] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 02a5a9f6601045b6b339c43c526fc29e) I0108 16:52:23.755795 30508 raft_consensus.cc:499] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.755856 30508 leader_election.cc:293] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) W0108 16:52:23.756024 24434 leader_election.cc:339] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) I0108 16:52:23.756034 30309 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "2c3d9174c33442d28f28f4d0cda9fc51" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 376212 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" is_pre_election: true I0108 16:52:23.756059 30309 raft_consensus.cc:2396] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e61264bf360a4a62a3a5e3b9f2875f5b in term 1. I0108 16:52:23.756117 24432 leader_election.cc:307] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: 02a5a9f6601045b6b339c43c526fc29e I0108 16:52:23.756186 30508 raft_consensus.cc:2696] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Leader pre-election won for term 2 I0108 16:52:23.756201 30508 raft_consensus.cc:477] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 02a5a9f6601045b6b339c43c526fc29e) I0108 16:52:23.756207 30508 raft_consensus.cc:2949] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 1 FOLLOWER]: Advancing to term 2 I0108 16:52:23.756814 30508 raft_consensus.cc:499] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.756855 30508 leader_election.cc:293] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 election: Requested vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), 72e6f176048945dea46e0f4305315d3c (127.23.197.131:49342) I0108 16:52:23.756978 30309 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "2c3d9174c33442d28f28f4d0cda9fc51" candidate_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 376212 } } ignore_live_leader: false dest_uuid: "72e6f176048945dea46e0f4305315d3c" W0108 16:52:23.756985 24434 leader_election.cc:339] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) I0108 16:52:23.756997 30309 raft_consensus.cc:2949] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 1 FOLLOWER]: Advancing to term 2 I0108 16:52:23.761560 30309 raft_consensus.cc:2396] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61264bf360a4a62a3a5e3b9f2875f5b in term 2. I0108 16:52:23.761699 24432 leader_election.cc:307] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: 02a5a9f6601045b6b339c43c526fc29e I0108 16:52:23.761750 30508 raft_consensus.cc:2696] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Leader election won for term 2 I0108 16:52:23.761801 30508 raft_consensus.cc:672] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 LEADER]: Becoming Leader. State: Replica: e61264bf360a4a62a3a5e3b9f2875f5b, State: Running, Role: LEADER I0108 16:52:23.761855 30508 consensus_queue.cc:206] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 376212, Committed index: 376212, Last appended: 1.376212, Last appended by leader: 376212, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.762233 24372 catalog_manager.cc:4143] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b reported cstate change: term changed from 1 to 2, leader changed from 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130) to e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129). New cstate: current_term: 2 leader_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } health_report { overall_health: UNKNOWN } } } I0108 16:52:23.962576 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.036414 I0108 16:52:23.963601 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.002s Metrics: {"bytes_written":438,"delete_count":0,"lbm_write_time_us":39,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1} I0108 16:52:23.963683 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.282490 I0108 16:52:23.964653 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_cache_hit":5,"cfile_cache_hit_bytes":2970,"cfile_cache_miss":2,"cfile_cache_miss_bytes":337,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":17,"lbm_reads_lt_1ms":10,"lbm_write_time_us":42,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":73,"threads_started":2,"update_count":2} I0108 16:52:23.964726 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000 I0108 16:52:23.964793 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {} I0108 16:52:23.974647 30530 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 02a5a9f6601045b6b339c43c526fc29e) I0108 16:52:23.974779 30530 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.975057 30530 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094) W0108 16:52:23.975116 30241 leader_election.cc:339] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) I0108 16:52:23.981951 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "72e6f176048945dea46e0f4305315d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 368072 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" is_pre_election: true I0108 16:52:23.981979 24501 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72e6f176048945dea46e0f4305315d3c in term 2. I0108 16:52:23.982112 30239 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: 02a5a9f6601045b6b339c43c526fc29e I0108 16:52:23.982203 30530 raft_consensus.cc:2696] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Leader pre-election won for term 3 I0108 16:52:23.982244 30530 raft_consensus.cc:477] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 02a5a9f6601045b6b339c43c526fc29e) I0108 16:52:23.982281 30530 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Advancing to term 3 I0108 16:52:23.983570 30530 raft_consensus.cc:499] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.983660 30530 leader_election.cc:293] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 3 election: Requested vote from peers 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520), e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094) I0108 16:52:23.983834 24501 tablet_service.cc:1319] Received RequestConsensusVote() RPC: tablet_id: "97a6791627c941a48f3269f2d00d1eea" candidate_uuid: "72e6f176048945dea46e0f4305315d3c" candidate_term: 3 candidate_status { last_received { term: 2 index: 368072 } } ignore_live_leader: false dest_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" I0108 16:52:23.983846 24501 raft_consensus.cc:2949] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 2 FOLLOWER]: Advancing to term 3 W0108 16:52:23.983914 30241 leader_election.cc:339] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) I0108 16:52:23.984884 24501 raft_consensus.cc:2396] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72e6f176048945dea46e0f4305315d3c in term 3. I0108 16:52:23.984988 30239 leader_election.cc:307] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72e6f176048945dea46e0f4305315d3c, e61264bf360a4a62a3a5e3b9f2875f5b; no voters: 02a5a9f6601045b6b339c43c526fc29e I0108 16:52:23.985044 30530 raft_consensus.cc:2696] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 3 FOLLOWER]: Leader election won for term 3 I0108 16:52:23.985086 30530 raft_consensus.cc:672] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [term 3 LEADER]: Becoming Leader. State: Replica: 72e6f176048945dea46e0f4305315d3c, State: Running, Role: LEADER I0108 16:52:23.985141 30530 consensus_queue.cc:206] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368072, Committed index: 368072, Last appended: 2.368072, Last appended by leader: 368072, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } } I0108 16:52:23.985605 24372 catalog_manager.cc:4143] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c reported cstate change: term changed from 2 to 3, leader changed from 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130) to 72e6f176048945dea46e0f4305315d3c (127.23.197.131). New cstate: current_term: 3 leader_uuid: "72e6f176048945dea46e0f4305315d3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02a5a9f6601045b6b339c43c526fc29e" member_type: VOTER last_known_addr { host: "127.23.197.130" port: 37520 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 } health_report { overall_health: HEALTHY } } } I0108 16:52:24.297454 30309 raft_consensus.cc:1195] T 2c3d9174c33442d28f28f4d0cda9fc51 P 72e6f176048945dea46e0f4305315d3c [term 2 FOLLOWER]: Refusing update from remote peer e61264bf360a4a62a3a5e3b9f2875f5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 376212. Preceding OpId from leader: term: 2 index: 376213. (index mismatch) I0108 16:52:24.297677 30508 consensus_queue.cc:982] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72e6f176048945dea46e0f4305315d3c" member_type: VOTER last_known_addr { host: "127.23.197.131" port: 49342 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376213, Last known committed idx: 376212, Time since last communication: 0.000s W0108 16:52:24.298645 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. W0108 16:52:24.432978 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I0108 16:52:24.555532 24501 raft_consensus.cc:1195] T 97a6791627c941a48f3269f2d00d1eea P e61264bf360a4a62a3a5e3b9f2875f5b [term 3 FOLLOWER]: Refusing update from remote peer 72e6f176048945dea46e0f4305315d3c: Log matching property violated. Preceding OpId in replica: term: 2 index: 368072. Preceding OpId from leader: term: 3 index: 368073. (index mismatch) I0108 16:52:24.555719 30533 consensus_queue.cc:982] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" member_type: VOTER last_known_addr { host: "127.23.197.129" port: 35094 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 368073, Last known committed idx: 368072, Time since last communication: 0.000s W0108 16:52:25.225766 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0108 16:52:26.977618 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0108 16:52:27.080778 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. W0108 16:52:27.684895 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0108 16:52:29.266008 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0108 16:52:29.814844 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry. W0108 16:52:30.238570 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0108 16:52:31.967846 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0108 16:52:32.465323 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry. W0108 16:52:32.992816 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0108 16:52:34.547598 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0108 16:52:35.041127 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry. W0108 16:52:35.635586 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0108 16:52:37.052645 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0108 16:52:37.694082 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry. W0108 16:52:38.239547 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0108 16:52:39.705646 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. I0108 16:52:39.934165 24342 meta_cache.cc:1038] Marking tablet server 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) as failed. W0108 16:52:39.934201 24342 meta_cache.cc:234] Tablet 2c3d9174c33442d28f28f4d0cda9fc51: Replica 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) W0108 16:52:40.040351 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry. W0108 16:52:40.673295 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0108 16:52:42.116772 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0108 16:52:42.724953 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry. W0108 16:52:43.121621 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. I0108 16:52:43.221333 24342 linked_list-test-util.h:876] Done collecting results (7989850 rows in 20610.7ms) I0108 16:52:44.144824 24342 linked_list-test-util.h:667] Snapshot: P: 1578530220348799 usec, L: 1 verified. Result: OK I0108 16:52:44.145334 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530520348549 usec, L: 1 Expected Rows: 15130600 I0108 16:52:44.148053 24342 linked_list-test-util.h:639] Calling callback on tserver e61264bf360a4a62a3a5e3b9f2875f5b W0108 16:52:44.481984 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0108 16:52:45.229532 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry. W0108 16:52:45.690310 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0108 16:52:46.825126 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0108 16:52:47.844319 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry. W0108 16:52:48.237093 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0108 16:52:49.404584 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0108 16:52:50.610685 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry. W0108 16:52:50.749205 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0108 16:52:51.987645 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0108 16:52:53.181573 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry. W0108 16:52:53.231058 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0108 16:52:54.282678 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0108 16:52:55.500150 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry. W0108 16:52:55.711942 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0108 16:52:56.732903 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0108 16:52:57.872364 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry. W0108 16:52:58.350234 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0108 16:52:59.267181 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0108 16:53:00.339996 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry. W0108 16:53:00.835189 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0108 16:53:01.860272 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0108 16:53:02.893746 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry. W0108 16:53:03.159413 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0108 16:53:04.171746 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0108 16:53:05.533839 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry. W0108 16:53:05.656839 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0108 16:53:06.761107 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. I0108 16:53:07.149879 24342 meta_cache.cc:1038] Marking tablet server 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) as failed. W0108 16:53:07.149907 24342 meta_cache.cc:234] Tablet 2c3d9174c33442d28f28f4d0cda9fc51: Replica 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520) has failed: Network error: TS failed: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111) W0108 16:53:07.922511 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry. W0108 16:53:08.446714 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0108 16:53:09.353386 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0108 16:53:10.498818 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry. W0108 16:53:11.195770 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0108 16:53:11.941941 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0108 16:53:13.247668 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry. W0108 16:53:13.798341 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0108 16:53:14.577512 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0108 16:53:15.653825 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry. W0108 16:53:16.185293 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0108 16:53:16.861996 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. W0108 16:53:18.245342 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry. I0108 16:53:18.458618 24342 linked_list-test-util.h:876] Done collecting results (15130600 rows in 34310.6ms) W0108 16:53:18.752621 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. W0108 16:53:19.554688 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. I0108 16:53:20.303339 24342 linked_list-test-util.h:667] Snapshot: P: 1578530520348549 usec, L: 1 verified. Result: OK I0108 16:53:20.304026 24342 linked_list-test-util.h:620] Verifying Snapshot: P: 1578530820374877 usec, L: 1 Expected Rows: 18767550 I0108 16:53:20.304992 24342 linked_list-test-util.h:639] Calling callback on tserver 72e6f176048945dea46e0f4305315d3c W0108 16:53:20.492522 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry. W0108 16:53:21.135641 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0108 16:53:21.995692 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0108 16:53:22.966763 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry. W0108 16:53:23.595132 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0108 16:53:24.342589 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0108 16:53:25.358054 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry. W0108 16:53:26.233948 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0108 16:53:27.202402 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0108 16:53:27.741557 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry. W0108 16:53:28.667577 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0108 16:53:29.561815 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0108 16:53:30.190102 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry. W0108 16:53:31.020598 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0108 16:53:31.857914 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0108 16:53:32.783870 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry. W0108 16:53:33.461169 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0108 16:53:34.475944 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0108 16:53:35.207602 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry. W0108 16:53:36.205355 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0108 16:53:36.986233 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0108 16:53:37.668637 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry. W0108 16:53:38.911986 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0108 16:53:39.252180 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0108 16:53:40.312952 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry. W0108 16:53:41.501058 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0108 16:53:41.846839 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0108 16:53:42.684653 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry. W0108 16:53:44.176316 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0108 16:53:44.481267 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0108 16:53:45.261548 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry. W0108 16:53:46.677371 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0108 16:53:46.918246 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0108 16:53:47.705720 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry. W0108 16:53:49.232933 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0108 16:53:49.305003 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0108 16:53:50.379604 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry. W0108 16:53:51.669615 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0108 16:53:51.922997 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0108 16:53:53.058526 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry. W0108 16:53:54.316473 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0108 16:53:54.548501 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. W0108 16:53:55.483737 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry. I0108 16:53:55.601763 24342 linked_list-test-util.h:876] Done collecting results (18767550 rows in 35296.8ms) W0108 16:53:56.929531 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0108 16:53:57.007059 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. I0108 16:53:57.913856 24342 linked_list-test-util.h:667] Snapshot: P: 1578530820374877 usec, L: 1 verified. Result: OK I0108 16:53:57.914470 24342 linked_list-test-util.h:620] Verifying Snapshot: NOW Expected Rows: 18767550 W0108 16:53:58.117866 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry. W0108 16:53:59.521692 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0108 16:53:59.677456 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0108 16:54:00.431929 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry. W0108 16:54:01.918754 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0108 16:54:02.170683 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0108 16:54:02.997087 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry. W0108 16:54:04.454782 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0108 16:54:04.752636 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0108 16:54:05.530807 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry. W0108 16:54:07.021734 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0108 16:54:07.060459 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0108 16:54:07.707267 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry. W0108 16:54:09.477458 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0108 16:54:09.563957 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0108 16:54:10.160001 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry. W0108 16:54:11.702342 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0108 16:54:12.214789 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0108 16:54:12.706818 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry. W0108 16:54:14.325600 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0108 16:54:14.712957 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0108 16:54:15.092777 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry. W0108 16:54:16.891798 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0108 16:54:16.979043 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0108 16:54:17.662813 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry. W0108 16:54:19.348937 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0108 16:54:19.764626 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0108 16:54:20.346666 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry. W0108 16:54:22.115350 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0108 16:54:22.176512 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. W0108 16:54:23.053687 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry. I0108 16:54:24.050088 24369 maintenance_manager.cc:326] P dc13e8f375d24e5a866a0ef1632607ea: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033357 I0108 16:54:24.050933 24368 maintenance_manager.cc:527] P dc13e8f375d24e5a866a0ef1632607ea: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.001s Metrics: {"bytes_written":438,"delete_count":0,"lbm_write_time_us":34,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":46,"threads_started":1,"update_count":1,"wal-append.queue_time_us":72} W0108 16:54:24.583470 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0108 16:54:24.808215 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0108 16:54:25.364176 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry. W0108 16:54:27.204267 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0108 16:54:27.379573 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0108 16:54:27.969619 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry. W0108 16:54:29.658648 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0108 16:54:29.731276 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0108 16:54:30.387261 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry. W0108 16:54:32.478626 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0108 16:54:32.510453 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. W0108 16:54:32.765265 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry. I0108 16:54:33.264267 24342 linked_list-test-util.h:876] Done collecting results (18767550 rows in 34954.7ms) W0108 16:54:35.069387 30241 consensus_peers.cc:473] T 97a6791627c941a48f3269f2d00d1eea P 72e6f176048945dea46e0f4305315d3c -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0108 16:54:35.160948 24434 consensus_peers.cc:473] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. W0108 16:54:35.169659 24434 consensus_peers.cc:473] T c0c87b04bc0f4f37876458fbfb4bdf03 P e61264bf360a4a62a3a5e3b9f2875f5b -> Peer 02a5a9f6601045b6b339c43c526fc29e (127.23.197.130:37520): Couldn't send request to peer 02a5a9f6601045b6b339c43c526fc29e. Status: Network error: Client connection negotiation failed: client connection to 127.23.197.130:37520: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry. I0108 16:54:35.559263 24342 linked_list-test-util.h:667] Snapshot: NOW verified. Result: OK I0108 16:54:35.559456 24342 linked_list-test-util.h:801] Successfully verified 18767550 rows I0108 16:54:35.559499 24342 linked_list-test.cc:215] Done with tserver kill test. I0108 16:54:35.562736 24342 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 24414 I0108 16:54:35.604497 24342 external_mini_cluster.cc:1241] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver with pid 30224 I0108 16:54:35.643009 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.23.197.129:35094 --local_ip_for_outbound_sockets=127.23.197.129 --webserver_port=37729 --webserver_interface=127.23.197.129 --tserver_master_addrs=127.23.197.190:59202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:54:35.655248 30848 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:54:35.655355 30848 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:54:35.656152 30848 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:54:35.656183 30848 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.197.129 I0108 16:54:35.657202 30848 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.197.129:35094 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/info.pb --webserver_interface=127.23.197.129 --webserver_port=37729 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.197.190:59202 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.30848 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.197.129 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:54:35.659327 30848 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:54:35.660455 30848 hybrid_clock.cc:186] HybridClock initialized: now 1578531275660445 us; error 15 us; skew 500 ppm I0108 16:54:35.660537 30848 fs_manager.cc:263] Metadata directory not provided I0108 16:54:35.660565 30848 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:54:35.662744 30848 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s I0108 16:54:35.662899 30848 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:54:35.662966 30848 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:54:35.665030 30848 fs_manager.cc:434] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s I0108 16:54:35.665216 30848 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal uuid: "e61264bf360a4a62a3a5e3b9f2875f5b" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:54:35.665249 30848 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/data Total live blocks: 66 Total live bytes: 733679791 Total live bytes (after alignment): 733827072 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:54:35.678758 30848 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:54:35.678970 30848 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0108 16:54:35.679790 30848 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0108 16:54:35.679810 30848 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0108 16:54:35.680102 30870 tablet_bootstrap.cc:439] T 2c3d9174c33442d28f28f4d0cda9fc51 P e61264bf360a4a62a3a5e3b9f2875f5b: Bootstrap starting. I0108 16:54:35.680344 30848 ts_tablet_manager.cc:415] Registered 3 tablets I0108 16:54:35.689203 30848 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.129:35094 I0108 16:54:35.689512 30848 webserver.cc:382] Webserver started at http://127.23.197.129:37729/ using document root and password file I0108 16:54:35.689668 30848 server_base.cc:629] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-0/data/info.pb I0108 16:54:35.693104 30975 heartbeater.cc:322] Connected to a master server at 127.23.197.190:59202 I0108 16:54:35.693122 30975 heartbeater.cc:406] Registering TS with master... I0108 16:54:35.693156 30975 heartbeater.cc:455] Master 127.23.197.190:59202 requested a full tablet report, sending... I0108 16:54:35.693334 24372 ts_manager.cc:192] Re-registered known tserver with Master: e61264bf360a4a62a3a5e3b9f2875f5b (127.23.197.129:35094) I0108 16:54:35.694146 24372 master_service.cc:351] Signed X509 certificate for tserver {username='jenkins'} at 127.23.197.129:56716 I0108 16:54:35.696250 24342 external_mini_cluster.cc:1105] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver as pid 30848 I0108 16:54:35.696460 24342 external_mini_cluster.cc:1043] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/release/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.23.197.130:37520 --local_ip_for_outbound_sockets=127.23.197.130 --webserver_port=44676 --webserver_interface=127.23.197.130 --tserver_master_addrs=127.23.197.190:59202 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.23.197.148:40982 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir --tablet_history_max_age_sec=86400 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0108 16:54:35.709375 30979 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0108 16:54:35.709502 30979 flags.cc:404] Enabled unsafe flag: --never_fsync=true W0108 16:54:35.710306 30979 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W0108 16:54:35.710337 30979 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.197.130 I0108 16:54:35.711371 30979 tablet_server_main.cc:76] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.23.197.148:40982 --ntp_initial_sync_wait_secs=10 --time_source=builtin --skip_remove_old_recovery_dir=true --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.23.197.130:37520 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/info.pb --webserver_interface=127.23.197.130 --webserver_port=44676 --tablet_history_max_age_sec=86400 --tserver_master_addrs=127.23.197.190:59202 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.30979 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.23.197.130 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 69d4e8116a928cef556b5447e30f4b0ed4ea0978 build type RELEASE built by jenkins at 08 Jan 2020 16:19:18 PST on quasar-xuwpic-1.vpc.cloudera.com I0108 16:54:35.713404 30979 hybrid_clock.cc:168] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I0108 16:54:35.714591 30979 hybrid_clock.cc:186] HybridClock initialized: now 1578531275714580 us; error 18 us; skew 500 ppm I0108 16:54:35.714674 30979 fs_manager.cc:263] Metadata directory not provided I0108 16:54:35.714704 30979 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0108 16:54:35.716868 30979 fs_manager.cc:414] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s I0108 16:54:35.717023 30979 env_posix.cc:1682] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0108 16:54:35.717089 30979 file_cache.cc:465] Constructed file cache lbm with capacity 13106 I0108 16:54:35.719563 30979 fs_manager.cc:434] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0108 16:54:35.719746 30979 fs_manager.cc:451] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal uuid: "02a5a9f6601045b6b339c43c526fc29e" format_stamp: "Formatted at 2020-01-09 00:31:59 on quasar-xuwpic-1.vpc.cloudera.com" I0108 16:54:35.719779 30979 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/linked_list-test.LinkedListTest.TestLoadAndVerify.1578529919787104-24342/linked-list-cluster/ts-1/data/data Total live blocks: 72 Total live bytes: 733674934 Total live bytes (after alignment): 733839360 Total number of LBM containers: 7 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0108 16:54:35.734051 30979 env_posix.cc:1677] Raising this process' running threads per effective uid limit from 32768 to 65536 I0108 16:54:35.734261 30979 ts_tablet_manager.cc:383] Loading tablet metadata (0/3 complete) I0108 16:54:35.735086 30979 ts_tablet_manager.cc:395] Loaded tablet metadata (3 total tablets, 3 live tablets) I0108 16:54:35.735105 30979 ts_tablet_manager.cc:401] Registering tablets (0/3 complete) I0108 16:54:35.735384 31002 tablet_bootstrap.cc:439] T 97a6791627c941a48f3269f2d00d1eea P 02a5a9f6601045b6b339c43c526fc29e: Bootstrap starting. I0108 16:54:35.735843 30979 ts_tablet_manager.cc:415] Registered 3 tablets I0108 16:54:35.744769 30979 rpc_server.cc:211] RPC server started. Bound to: 127.23.197.130:37520 E0108 16:54:35.744949 30979 webserver.cc:443] Webserver: set_ports_option: cannot bind to 127.23.197.130:44676: 98 (Address already in use) W0108 16:54:35.744971 30979 net_util.cc:463] Trying to use lsof to find any processes listening on 0.0.0.0:44676 I0108 16:54:35.744977 30979 net_util.cc:466] $ export PATH=$PATH:/usr/sbin ; lsof -n -i 'TCP:44676' -sTCP:LISTEN ; for pid in $(lsof -F p -n -i 'TCP:44676' -sTCP:LISTEN | grep p | cut -f 2 -dp) ; do while [ $pid -gt 1 ] ; do ps h -fp $pid ; stat=($(