[==========] Running 1 test from 1 test case. [----------] Global test environment set-up. [----------] 1 test from UpdateScanDeltaCompactionTest [ RUN ] UpdateScanDeltaCompactionTest.TestAll I1118 00:57:13.606796 14341 system_ntp.cc:78] NTP initialized. Skew: 500ppm Current error: 91801us I1118 00:57:13.607641 14341 server_base.cc:229] Could not load existing FS layout: Not found: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/master-0-root/instance: No such file or directory (error 2) I1118 00:57:13.607720 14341 server_base.cc:230] Creating new FS layout I1118 00:57:13.612555 14341 fs_manager.cc:565] Generated new instance metadata in path /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/master-0-root/instance: uuid: "a06d32a684114b489db532b6b72c83ea" format_stamp: "Formatted at 2017-11-18 00:57:13 on dist-test-slave-dist-test-slave-9h4n" I1118 00:57:13.620790 14341 fs_manager.cc:469] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s I1118 00:57:13.621665 14341 env_posix.cc:1596] Not raising process file limit of 65536; it is already as high as it can go I1118 00:57:13.621928 14341 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I1118 00:57:13.626446 14341 fs_manager.cc:382] Time spent opening block manager: real 0.002s user 0.000s sys 0.004s I1118 00:57:13.626585 14341 fs_manager.cc:402] Opened local filesystem: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/master-0-root uuid: "a06d32a684114b489db532b6b72c83ea" format_stamp: "Formatted at 2017-11-18 00:57:13 on dist-test-slave-dist-test-slave-9h4n" I1118 00:57:13.626729 14341 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1118 00:57:13.652709 14355 process_memory.cc:180] Process hard memory limit is 11.777661 GB I1118 00:57:13.652762 14355 process_memory.cc:182] Process soft memory limit is 9.422129 GB I1118 00:57:13.652791 14355 process_memory.cc:185] Process memory pressure threshold is 7.066597 GB I1118 00:57:13.727823 14341 rpc_server.cc:192] RPC server started. Bound to: 127.14.1.126:59179 I1118 00:57:13.728389 14341 webserver.cc:173] Starting webserver on 127.14.1.126:0 I1118 00:57:13.729110 14341 webserver.cc:184] Document root disabled I1118 00:57:13.729780 14341 webserver.cc:311] Webserver started. Bound to: http://127.14.1.126:60856/ I1118 00:57:13.744012 14399 tablet_bootstrap.cc:433] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: Bootstrap starting. I1118 00:57:13.746971 14399 tablet_bootstrap.cc:568] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: No blocks or log segments found. Creating new log. I1118 00:57:13.747772 14399 log.cc:520] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: Log is configured to *not* fsync() on all Append() calls I1118 00:57:13.749097 14399 tablet_bootstrap.cc:433] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: No bootstrap required, opened a new log I1118 00:57:13.750574 14399 raft_consensus.cc:286] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d32a684114b489db532b6b72c83ea" member_type: VOTER } I1118 00:57:13.750810 14399 raft_consensus.cc:312] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1118 00:57:13.750871 14399 raft_consensus.cc:585] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a06d32a684114b489db532b6b72c83ea, State: Initialized, Role: FOLLOWER I1118 00:57:13.751322 14399 consensus_queue.cc:205] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER I1118 00:57:13.751428 14399 raft_consensus.cc:326] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1118 00:57:13.751482 14399 raft_consensus.cc:418] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1118 00:57:13.751565 14399 raft_consensus.cc:2501] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: Snoozing failure detection for 2.016s (starting election) I1118 00:57:13.751631 14399 raft_consensus.cc:2553] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 0 FOLLOWER]: Advancing to term 1 I1118 00:57:13.753329 14399 raft_consensus.cc:440] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d32a684114b489db532b6b72c83ea" member_type: VOTER } I1118 00:57:13.753728 14399 leader_election.cc:258] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1118 00:57:13.754329 14401 raft_consensus.cc:2501] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: Snoozing failure detection for 1.730s (election complete) I1118 00:57:13.754422 14401 raft_consensus.cc:2310] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 1 FOLLOWER]: Leader election won for term 1 I1118 00:57:13.757764 14401 raft_consensus.cc:552] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [term 1 LEADER]: Becoming Leader. State: Replica: a06d32a684114b489db532b6b72c83ea, State: Running, Role: LEADER I1118 00:57:13.757969 14401 consensus_queue.cc:184] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d32a684114b489db532b6b72c83ea" member_type: VOTER } I1118 00:57:13.766125 14403 sys_catalog.cc:329] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a06d32a684114b489db532b6b72c83ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d32a684114b489db532b6b72c83ea" member_type: VOTER } } I1118 00:57:13.766351 14403 sys_catalog.cc:332] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: This master's current role is: LEADER I1118 00:57:13.767354 14399 sys_catalog.cc:329] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a06d32a684114b489db532b6b72c83ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d32a684114b489db532b6b72c83ea" member_type: VOTER } } I1118 00:57:13.767678 14399 sys_catalog.cc:332] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: This master's current role is: LEADER I1118 00:57:13.767473 14402 sys_catalog.cc:329] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a06d32a684114b489db532b6b72c83ea. Latest consensus state: current_term: 1 leader_uuid: "a06d32a684114b489db532b6b72c83ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a06d32a684114b489db532b6b72c83ea" member_type: VOTER } } I1118 00:57:13.768719 14402 sys_catalog.cc:332] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: This master's current role is: LEADER I1118 00:57:13.770048 14399 sys_catalog.cc:416] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea [sys.catalog]: configured and running, proceeding with master startup. I1118 00:57:13.770625 14405 catalog_manager.cc:934] Loading table and tablet metadata into memory... I1118 00:57:13.777753 14405 catalog_manager.cc:945] Initializing Kudu internal certificate authority... I1118 00:57:13.812772 14405 catalog_manager.cc:832] Generated new certificate authority record I1118 00:57:13.814241 14405 catalog_manager.cc:954] Loading token signing keys... I1118 00:57:13.849722 14405 catalog_manager.cc:3563] T 00000000000000000000000000000000 P a06d32a684114b489db532b6b72c83ea: Generated new TSK 0 I1118 00:57:13.913568 14341 system_ntp.cc:78] NTP initialized. Skew: 500ppm Current error: 91801us I1118 00:57:13.914575 14341 server_base.cc:229] Could not load existing FS layout: Not found: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/ts-0-root/instance: No such file or directory (error 2) I1118 00:57:13.914669 14341 server_base.cc:230] Creating new FS layout I1118 00:57:13.917642 14341 fs_manager.cc:565] Generated new instance metadata in path /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/ts-0-root/instance: uuid: "94a8b1472dbd4981bcb4853362603824" format_stamp: "Formatted at 2017-11-18 00:57:13 on dist-test-slave-dist-test-slave-9h4n" I1118 00:57:13.933815 14341 fs_manager.cc:469] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s I1118 00:57:13.935434 14341 file_cache.cc:470] Constructed file cache lbm with capacity 26214 I1118 00:57:13.944828 14341 fs_manager.cc:382] Time spent opening block manager: real 0.007s user 0.000s sys 0.008s I1118 00:57:13.944973 14341 fs_manager.cc:402] Opened local filesystem: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/ts-0-root uuid: "94a8b1472dbd4981bcb4853362603824" format_stamp: "Formatted at 2017-11-18 00:57:13 on dist-test-slave-dist-test-slave-9h4n" I1118 00:57:13.945097 14341 fs_report.cc:347] Block manager report -------------------- 1 data directories: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1118 00:57:14.021453 14341 ts_tablet_manager.cc:296] Loaded tablet metadata (0 live tablets) I1118 00:57:14.182675 14341 rpc_server.cc:192] RPC server started. Bound to: 127.14.1.65:46405 I1118 00:57:14.183017 14341 webserver.cc:173] Starting webserver on 127.14.1.65:0 I1118 00:57:14.183043 14341 webserver.cc:184] Document root disabled I1118 00:57:14.183496 14341 webserver.cc:311] Webserver started. Bound to: http://127.14.1.65:54655/ I1118 00:57:14.202373 14473 heartbeater.cc:324] Connected to a master server at 127.14.1.126:59179 I1118 00:57:14.202528 14473 heartbeater.cc:401] Registering TS with master... I1118 00:57:14.202813 14473 heartbeater.cc:428] Master 127.14.1.126:59179 requested a full tablet report, sending... I1118 00:57:14.203949 14366 ts_manager.cc:81] Registered new tserver with Master: 94a8b1472dbd4981bcb4853362603824 (127.14.1.65:46405) I1118 00:57:14.205706 14366 master_service.cc:205] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41247 I1118 00:57:14.206554 14341 internal_mini_cluster.cc:304] 1 TS(s) registered with all masters after 0.0225685s I1118 00:57:14.230242 14366 catalog_manager.cc:1197] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41248: name: "update-scan-delta-compact-tbl" schema { columns { name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int64" type: INT64 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I1118 00:57:14.262892 14440 tablet_service.cc:752] Processing CreateTablet for tablet d0d2ada227b049d4a947d8f2ef96adc0 (table=update-scan-delta-compact-tbl [id=b65dfde32eb241ccab90605b51082289]), partition=RANGE (key) PARTITION UNBOUNDED I1118 00:57:14.266782 14440 ts_tablet_manager.cc:1067] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Registered tablet (data state: TABLET_DATA_READY) I1118 00:57:14.267204 14485 ts_tablet_manager.cc:917] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Bootstrapping tablet I1118 00:57:14.267998 14485 tablet_bootstrap.cc:433] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Bootstrap starting. I1118 00:57:14.269855 14485 tablet_bootstrap.cc:568] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: No blocks or log segments found. Creating new log. I1118 00:57:14.271157 14485 tablet_bootstrap.cc:433] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: No bootstrap required, opened a new log I1118 00:57:14.271271 14485 ts_tablet_manager.cc:942] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1118 00:57:14.271766 14485 raft_consensus.cc:286] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a8b1472dbd4981bcb4853362603824" member_type: VOTER last_known_addr { host: "127.14.1.65" port: 46405 } } I1118 00:57:14.271945 14485 raft_consensus.cc:312] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1118 00:57:14.271992 14485 raft_consensus.cc:585] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a8b1472dbd4981bcb4853362603824, State: Initialized, Role: FOLLOWER I1118 00:57:14.272137 14485 consensus_queue.cc:205] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER I1118 00:57:14.272215 14485 raft_consensus.cc:326] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1118 00:57:14.272264 14485 raft_consensus.cc:418] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1118 00:57:14.272313 14485 raft_consensus.cc:2501] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Snoozing failure detection for 1.584s (starting election) I1118 00:57:14.272369 14485 raft_consensus.cc:2553] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 0 FOLLOWER]: Advancing to term 1 I1118 00:57:14.273859 14485 raft_consensus.cc:440] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a8b1472dbd4981bcb4853362603824" member_type: VOTER last_known_addr { host: "127.14.1.65" port: 46405 } } I1118 00:57:14.274682 14485 leader_election.cc:258] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I1118 00:57:14.275104 14487 raft_consensus.cc:2501] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Snoozing failure detection for 2.159s (election complete) I1118 00:57:14.275182 14487 raft_consensus.cc:2310] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 1 FOLLOWER]: Leader election won for term 1 I1118 00:57:14.281509 14487 raft_consensus.cc:552] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [term 1 LEADER]: Becoming Leader. State: Replica: 94a8b1472dbd4981bcb4853362603824, State: Running, Role: LEADER I1118 00:57:14.281590 14487 consensus_queue.cc:184] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a8b1472dbd4981bcb4853362603824" member_type: VOTER last_known_addr { host: "127.14.1.65" port: 46405 } } I1118 00:57:14.282045 14473 heartbeater.cc:420] Master 127.14.1.126:59179 was elected leader, sending a full tablet report... I1118 00:57:14.285487 14485 ts_tablet_manager.cc:970] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Time spent starting tablet: real 0.014s user 0.000s sys 0.012s I1118 00:57:14.286746 14366 catalog_manager.cc:3340] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824 reported cstate change: term changed from 0 to 1, leader changed from to 94a8b1472dbd4981bcb4853362603824 (127.14.1.65). New cstate: current_term: 1 leader_uuid: "94a8b1472dbd4981bcb4853362603824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a8b1472dbd4981bcb4853362603824" member_type: VOTER last_known_addr { host: "127.14.1.65" port: 46405 } } } I1118 00:57:20.185775 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:57:20.185827 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:57:20.185870 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:57:20.185887 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 10362647 bytes I1118 00:57:20.185925 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919358028165120 or (T in {6188919358028165120})}] I1118 00:57:20.191184 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:57:23.428817 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:57:23.462417 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:57:23.462544 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919371363065856 or (T in {6188919371363065856})}] I1118 00:57:25.245332 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 75001 rows (200224 bytes) I1118 00:57:25.247941 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 5.063s user 2.604s sys 0.216s I1118 00:57:25.248024 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"compiler_manager_pool.queue_time_us":254,"compiler_manager_pool.run_cpu_time_us":295847,"compiler_manager_pool.run_wall_time_us":483129,"data dirs.queue_time_us":47586,"data dirs.run_cpu_time_us":214,"data dirs.run_wall_time_us":148850,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":149784,"lbm_writes_1-10_ms":1,"lbm_writes_gt_100_ms":1,"lbm_writes_lt_1ms":104,"thread_start_us":47732,"threads_started":2} I1118 00:57:25.252128 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:57:25.252190 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4} I1118 00:57:27.269011 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:57:27.269050 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:57:27.269060 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:57:27.269073 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 11403031 bytes I1118 00:57:27.269299 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919387105611776 or (T in {6188919387105611776})}] I1118 00:57:27.270297 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:57:30.970187 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:57:30.973268 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:57:30.973331 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919402258792448 or (T in {6188919402258792448})}] I1118 00:57:31.814748 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 84000 rows (226054 bytes) I1118 00:57:31.819249 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 4.553s user 2.752s sys 0.060s I1118 00:57:31.819334 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":21280,"data dirs.run_cpu_time_us":197,"data dirs.run_wall_time_us":1557,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6053,"lbm_writes_lt_1ms":116,"spinlock_wait_cycles":427904,"thread_start_us":21181,"threads_started":1} I1118 00:57:31.820617 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:57:31.820677 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4} I1118 00:57:33.821543 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:57:33.821596 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:57:33.821607 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:57:33.821619 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 10362647 bytes I1118 00:57:33.821640 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919413922365440 or (T in {6188919413922365440})}] I1118 00:57:33.822661 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:57:36.415030 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:57:36.441206 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:57:36.441300 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919424591011840 or (T in {6188919424591011840})}] I1118 00:57:37.287533 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 73000 rows (195805 bytes) I1118 00:57:37.291652 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 3.470s user 2.592s sys 0.008s I1118 00:57:37.291754 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6245,"data dirs.run_cpu_time_us":196,"data dirs.run_wall_time_us":8012,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":41090,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":102,"thread_start_us":6149,"threads_started":1} I1118 00:57:37.292937 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:57:37.292999 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I1118 00:57:41.317951 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:57:41.317996 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:57:41.318008 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:57:41.318020 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 12443415 bytes I1118 00:57:41.318239 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919444648693760 or (T in {6188919444648693760})}] I1118 00:57:41.319381 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:57:43.934671 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:57:43.947536 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:57:43.947615 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919455317164032 or (T in {6188919455317164032})}] I1118 00:57:44.785209 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 87000 rows (232723 bytes) I1118 00:57:44.790290 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 3.496s user 3.164s sys 0.020s I1118 00:57:44.790380 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6249,"data dirs.run_cpu_time_us":183,"data dirs.run_wall_time_us":1872,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6047,"lbm_writes_lt_1ms":118,"spinlock_wait_cycles":413952,"thread_start_us":6145,"threads_started":1} I1118 00:57:44.791582 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:57:44.791651 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I1118 00:57:48.797057 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:57:48.797101 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:57:48.797112 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:57:48.797123 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 11403031 bytes I1118 00:57:48.797382 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919475259641856 or (T in {6188919475259641856})}] I1118 00:57:48.798674 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:57:51.516829 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:57:51.517125 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:57:51.517223 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919486267260928 or (T in {6188919486267260928})}] I1118 00:57:52.343858 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 84000 rows (227380 bytes) I1118 00:57:52.346277 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 3.554s user 3.004s sys 0.008s I1118 00:57:52.346384 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6253,"data dirs.run_cpu_time_us":198,"data dirs.run_wall_time_us":10217,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5882,"lbm_writes_lt_1ms":116,"spinlock_wait_cycles":494464,"thread_start_us":6147,"threads_started":1} I1118 00:57:52.347638 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:57:52.347702 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I1118 00:57:56.348995 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:57:56.349041 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:57:56.349052 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:57:56.349063 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 12443415 bytes I1118 00:57:56.349086 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919506131349504 or (T in {6188919506131349504})}] I1118 00:57:56.350400 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:57:58.638614 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:57:58.638977 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:57:58.639040 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919515191648256 or (T in {6188919515191648256})}] I1118 00:57:59.487498 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 85000 rows (227436 bytes) I1118 00:57:59.489087 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 3.141s user 3.032s sys 0.012s I1118 00:57:59.489135 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6282,"data dirs.run_cpu_time_us":165,"data dirs.run_wall_time_us":21590,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5654,"lbm_writes_lt_1ms":116,"thread_start_us":6172,"threads_started":1} I1118 00:57:59.494096 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:57:59.494134 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4} I1118 00:58:03.495791 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:58:03.495836 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:58:03.495846 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:58:03.495857 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 10362647 bytes I1118 00:58:03.495880 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919535325827072 or (T in {6188919535325827072})}] I1118 00:58:03.497001 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:58:05.600136 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:58:05.604449 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:58:05.604521 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919543977508864 or (T in {6188919543977508864})}] I1118 00:58:06.567148 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 75000 rows (200814 bytes) I1118 00:58:06.571625 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 3.076s user 2.592s sys 0.028s I1118 00:58:06.571713 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6241,"data dirs.run_cpu_time_us":183,"data dirs.run_wall_time_us":2003,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5072,"lbm_writes_lt_1ms":106,"thread_start_us":6138,"threads_started":1} I1118 00:58:06.573218 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:58:06.573281 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I1118 00:58:10.574630 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:58:10.574676 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:58:10.574689 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:58:10.574700 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 11403031 bytes I1118 00:58:10.574724 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919564385300480 or (T in {6188919564385300480})}] I1118 00:58:10.575875 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:58:12.862056 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:58:12.862459 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:58:12.862529 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919573478412288 or (T in {6188919573478412288})}] I1118 00:58:13.639175 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 80000 rows (213502 bytes) I1118 00:58:13.641399 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 3.067s user 2.924s sys 0.012s I1118 00:58:13.641497 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6262,"data dirs.run_cpu_time_us":220,"data dirs.run_wall_time_us":25108,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5279,"lbm_writes_lt_1ms":111,"thread_start_us":6149,"threads_started":1} I1118 00:58:13.642895 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:58:13.642962 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4} I1118 00:58:17.644304 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:58:17.644345 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:58:17.644356 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:58:17.644374 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 11403031 bytes I1118 00:58:17.644395 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919593333739520 or (T in {6188919593333739520})}] I1118 00:58:17.645455 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:58:20.726528 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:58:20.727074 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:58:20.727144 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919605707587584 or (T in {6188919605707587584})}] I1118 00:58:21.709559 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 80000 rows (213591 bytes) I1118 00:58:21.711748 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 4.068s user 2.280s sys 0.008s I1118 00:58:21.711838 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":5283,"data dirs.run_cpu_time_us":70,"data dirs.run_wall_time_us":70,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5339,"lbm_writes_lt_1ms":111,"thread_start_us":15143,"threads_started":1} I1118 00:58:21.713294 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:58:21.713352 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4} I1118 00:58:23.737129 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:58:23.737186 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:58:23.737198 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:58:23.737210 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 9322263 bytes I1118 00:58:23.737429 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919618379677696 or (T in {6188919618379677696})}] I1118 00:58:23.739284 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:58:25.577175 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:58:25.577666 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:58:25.577736 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919625478238208 or (T in {6188919625478238208})}] I1118 00:58:26.236054 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 64000 rows (172427 bytes) I1118 00:58:26.236248 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 2.522s user 2.388s sys 0.016s I1118 00:58:26.236343 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6263,"data dirs.run_cpu_time_us":224,"data dirs.run_wall_time_us":13845,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6458,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":90,"spinlock_wait_cycles":408320,"thread_start_us":6148,"threads_started":1} I1118 00:58:26.238018 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:58:26.238083 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4} I1118 00:58:30.239513 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:58:30.239558 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:58:30.239569 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:58:30.239581 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 10362647 bytes I1118 00:58:30.239604 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919644867579904 or (T in {6188919644867579904})}] I1118 00:58:30.240717 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:58:31.744889 14490 log.cc:644] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Max segment size reached. Starting new segment allocation I1118 00:58:31.791456 14490 log.cc:571] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Rolled over to a new log segment at /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/ts-0-root/wals/d0d2ada227b049d4a947d8f2ef96adc0/wal-000000002 I1118 00:58:32.299978 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:58:32.300500 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:58:32.300607 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919653073313792 or (T in {6188919653073313792})}] I1118 00:58:33.030443 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 74000 rows (200129 bytes) I1118 00:58:33.032475 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 2.793s user 2.712s sys 0.012s I1118 00:58:33.032557 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6213,"data dirs.run_cpu_time_us":192,"data dirs.run_wall_time_us":19014,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4818,"lbm_writes_lt_1ms":105,"thread_start_us":6121,"threads_started":1} I1118 00:58:33.034255 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:58:33.034322 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4} I1118 00:58:37.035915 14410 tablet.cc:1071] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering stage 1 (old memrowset already frozen for inserts) I1118 00:58:37.035959 14410 compaction.cc:903] Selected 1 rowsets to compact: I1118 00:58:37.035970 14410 compaction.cc:906] memrowset(current size on disk: ~0 bytes) I1118 00:58:37.035981 14410 tablet.cc:1073] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Memstore in-memory size: 11403031 bytes I1118 00:58:37.036003 14410 tablet.cc:1336] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6188919672770535424 or (T in {6188919672770535424})}] I1118 00:58:37.037086 14410 multi_column_writer.cc:103] Opened CFile writers for 3 column(s) I1118 00:58:39.335645 14410 tablet.cc:1421] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I1118 00:58:39.336192 14410 tablet.cc:1476] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush Phase 2: carrying over any updates which arrived during Phase 1 I1118 00:58:39.336264 14410 tablet.cc:1478] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6188919682043531264 or (T in {6188919682043531264})}] I1118 00:58:41.051676 14410 tablet.cc:1520] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Flush successful on 78000 rows (209435 bytes) I1118 00:58:41.067162 14410 maintenance_manager.cc:486] Time spent running FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0): real 4.032s user 2.872s sys 0.008s I1118 00:58:41.067262 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: FlushMRSOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"data dirs.queue_time_us":6236,"data dirs.run_cpu_time_us":161,"data dirs.run_wall_time_us":1592,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5179,"lbm_writes_lt_1ms":109,"thread_start_us":6142,"threads_started":1} I1118 00:58:41.068464 14410 log_reader.cc:386] T d0d2ada227b049d4a947d8f2ef96adc0: removed 1 log segments from log reader I1118 00:58:41.068568 14410 log.cc:856] T d0d2ada227b049d4a947d8f2ef96adc0 P 94a8b1472dbd4981bcb4853362603824: Deleting log segment in path: /tmp/run_tha_testRpEzQ8/test-tmp/update_scan_delta_compact-test.UpdateScanDeltaCompactionTest.TestAll.1510966633565201-14341/minicluster-data/ts-0-root/wals/d0d2ada227b049d4a947d8f2ef96adc0/wal-000000001 (ops 1-875) I1118 00:58:41.070516 14410 maintenance_manager.cc:486] Time spent running LogGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.002s user 0.000s sys 0.000s I1118 00:58:41.077024 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: LogGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {} I1118 00:58:41.078275 14410 maintenance_manager.cc:486] Time spent running UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0): real 0.000s user 0.000s sys 0.000s I1118 00:58:41.078331 14410 maintenance_manager.cc:492] P 94a8b1472dbd4981bcb4853362603824: UndoDeltaBlockGCOp(d0d2ada227b049d4a947d8f2ef96adc0) metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4} I1118 00:58:42.015146 14341 update_scan_delta_compact-test.cc:210] Time spent Insert: real 87.721s user 70.008s sys 1.244s W1118 00:58:47.072237 14479 meta_cache.cc:217] Tablet d0d2ada227b049d4a947d8f2ef96adc0: Replica 94a8b1472dbd4981bcb4853362603824 (127.14.1.65:46405) has failed: Timed out: Write RPC to 127.14.1.65:46405 timed out after 5.000s (SENT) W1118 00:58:47.072432 14479 batcher.cc:349] Timed out: Failed to write batch of 1000 ops to tablet d0d2ada227b049d4a947d8f2ef96adc0 after 1 attempt(s): Failed to write to server: 94a8b1472dbd4981bcb4853362603824 (127.14.1.65:46405): Write RPC to 127.14.1.65:46405 timed out after 5.000s (SENT) F1118 00:58:47.077445 14509 update_scan_delta_compact-test.cc:277] Check failed: _s.ok() Bad status: IO error: Some errors occurred *** Check failure stack trace: *** @ 0x7f17702cc62d google::LogMessage::Fail() at ??:0 @ 0x7f17702ce64c google::LogMessage::SendToLog() at ??:0 @ 0x7f17702cc189 google::LogMessage::Flush() at ??:0 @ 0x7f17702cefdf google::LogMessageFatal::~LogMessageFatal() at ??:0 @ 0x53649b kudu::tablet::UpdateScanDeltaCompactionTest::UpdateRows() at /home/jenkins-slave/workspace/kudu-master/3/src/kudu/integration-tests/update_scan_delta_compact-test.cc:276 (discriminator 7) @ 0x542035 boost::_bi::bind_t<>::operator()() at /home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1223 @ 0x7f177a57a6c0 boost::function0<>::operator()() at ??:0 @ 0x7f1773050a0f kudu::Thread::SuperviseThread() at ??:0 @ 0x7f1777834184 start_thread at ??:0 @ 0x7f176f420ffd clone at ??:0 @ (nil) (unknown)