[ RUN ] ClientTest.TestServerTooBusyRetry I0428 12:09:10.467747 8432 internal_mini_cluster.cc:146] Creating distributed mini masters. Addrs: 127.8.60.62:40873 I0428 12:09:10.477488 8432 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 104749us I0428 12:09:10.477797 8432 fs_manager.cc:263] Metadata directory not provided I0428 12:09:10.477906 8432 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0428 12:09:10.478090 8432 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0428 12:09:10.478183 8432 server_base.cc:434] Attempting to create new FS layout instead I0428 12:09:10.478911 8432 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/master-0-root/instance: uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" format_stamp: "Formatted at 2019-04-28 12:09:10 on dist-test-slave-dist-test-slave-lsq8" I0428 12:09:10.485749 8432 fs_manager.cc:503] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s I0428 12:09:10.486829 8432 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0428 12:09:10.492230 8432 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0428 12:09:10.492352 8432 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/master-0-root uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" format_stamp: "Formatted at 2019-04-28 12:09:10 on dist-test-slave-dist-test-slave-lsq8" I0428 12:09:10.492547 8432 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/master-0-root metadata directory: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/master-0-root 1 data directories: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/master-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0428 12:09:10.559653 8432 rpc_server.cc:205] RPC server started. Bound to: 127.8.60.62:40873 I0428 12:09:10.561096 8432 webserver.cc:308] Webserver started at http://127.8.60.62:34911/ using document root and password file I0428 12:09:10.562460 10187 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0428 12:09:10.569051 10187 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce: Bootstrap starting. I0428 12:09:10.570829 10187 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce: No blocks or log segments found. Creating new log. I0428 12:09:10.573029 10187 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce: No bootstrap required, opened a new log I0428 12:09:10.574060 10187 raft_consensus.cc:340] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } I0428 12:09:10.574337 10187 raft_consensus.cc:366] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0428 12:09:10.574453 10187 raft_consensus.cc:705] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7889a3f0d645a8aa308d11ba84d9ce, State: Initialized, Role: FOLLOWER I0428 12:09:10.574795 10187 consensus_queue.cc:229] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } I0428 12:09:10.575143 10187 raft_consensus.cc:380] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0428 12:09:10.575273 10187 raft_consensus.cc:472] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0428 12:09:10.575415 10187 raft_consensus.cc:2886] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 0 FOLLOWER]: Advancing to term 1 I0428 12:09:10.578245 10187 raft_consensus.cc:494] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } I0428 12:09:10.578577 10187 leader_election.cc:310] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee7889a3f0d645a8aa308d11ba84d9ce; no voters: I0428 12:09:10.579618 10187 leader_election.cc:296] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [CANDIDATE]: Term 1 election: Requested vote from peers I0428 12:09:10.579795 10189 raft_consensus.cc:2640] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 1 FOLLOWER]: Leader election won for term 1 I0428 12:09:10.581046 10189 raft_consensus.cc:667] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [term 1 LEADER]: Becoming Leader. State: Replica: ee7889a3f0d645a8aa308d11ba84d9ce, State: Running, Role: LEADER I0428 12:09:10.581369 10189 consensus_queue.cc:206] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } I0428 12:09:10.583063 10187 sys_catalog.cc:337] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } } I0428 12:09:10.583586 10187 sys_catalog.cc:340] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: This master's current role is: LEADER I0428 12:09:10.583242 10190 sys_catalog.cc:337] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } } I0428 12:09:10.583298 10191 sys_catalog.cc:337] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee7889a3f0d645a8aa308d11ba84d9ce. Latest consensus state: current_term: 1 leader_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7889a3f0d645a8aa308d11ba84d9ce" member_type: VOTER } } I0428 12:09:10.583870 10190 sys_catalog.cc:340] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: This master's current role is: LEADER I0428 12:09:10.584035 10191 sys_catalog.cc:340] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: This master's current role is: LEADER I0428 12:09:10.585456 10187 sys_catalog.cc:424] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce [sys.catalog]: configured and running, proceeding with master startup. I0428 12:09:10.585649 10193 catalog_manager.cc:1055] Loading table and tablet metadata into memory... I0428 12:09:10.587695 10193 catalog_manager.cc:1066] Initializing Kudu internal certificate authority... I0428 12:09:10.589948 8432 internal_mini_cluster.cc:166] Waiting to initialize catalog manager on master 0 I0428 12:09:10.611552 10193 catalog_manager.cc:950] Generated new certificate authority record I0428 12:09:10.612915 10193 catalog_manager.cc:1075] Loading token signing keys... I0428 12:09:10.626070 10193 catalog_manager.cc:4291] T 00000000000000000000000000000000 P ee7889a3f0d645a8aa308d11ba84d9ce: Generated new TSK 0 I0428 12:09:10.664409 8432 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 104749us I0428 12:09:10.664674 8432 fs_manager.cc:263] Metadata directory not provided I0428 12:09:10.664782 8432 fs_manager.cc:269] Using write-ahead log directory (fs_wal_dir) as metadata directory I0428 12:09:10.664921 8432 server_base.cc:433] Could not load existing FS layout: Not found: could not find a healthy instance file I0428 12:09:10.665021 8432 server_base.cc:434] Attempting to create new FS layout instead I0428 12:09:10.665721 8432 fs_manager.cc:602] Generated new instance metadata in path /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root/instance: uuid: "44b7784188f2497aabfce250a5c87ec4" format_stamp: "Formatted at 2019-04-28 12:09:10 on dist-test-slave-dist-test-slave-lsq8" I0428 12:09:10.672756 8432 fs_manager.cc:503] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s I0428 12:09:10.673847 8432 file_cache.cc:466] Constructed file cache lbm with capacity 419430 I0428 12:09:10.679373 8432 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0428 12:09:10.679493 8432 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root uuid: "44b7784188f2497aabfce250a5c87ec4" format_stamp: "Formatted at 2019-04-28 12:09:10 on dist-test-slave-dist-test-slave-lsq8" I0428 12:09:10.679678 8432 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0428 12:09:10.745239 8432 ts_tablet_manager.cc:356] Loaded tablet metadata (0 total tablets, 0 live tablets) I0428 12:09:10.745371 8432 ts_tablet_manager.cc:376] Registered 0 tablets I0428 12:09:10.779685 8432 rpc_server.cc:205] RPC server started. Bound to: 127.8.60.1:38987 I0428 12:09:10.781401 8432 webserver.cc:308] Webserver started at http://127.8.60.1:44795/ using document root and password file I0428 12:09:10.795549 10261 heartbeater.cc:345] Connected to a master server at 127.8.60.62:40873 I0428 12:09:10.795754 10261 heartbeater.cc:426] Registering TS with master... I0428 12:09:10.796098 10261 heartbeater.cc:475] Master 127.8.60.62:40873 requested a full tablet report, sending... I0428 12:09:10.797329 10154 ts_manager.cc:159] Registered new tserver with Master: 44b7784188f2497aabfce250a5c87ec4 (127.8.60.1:38987) I0428 12:09:10.797413 8432 internal_mini_cluster.cc:328] 1 TS(s) registered with all masters after 0.014440876s I0428 12:09:10.798986 10154 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51684 I0428 12:09:10.811384 10261 heartbeater.cc:467] Master 127.8.60.62:40873 was elected leader, sending a full tablet report... I0428 12:09:10.812574 10153 catalog_manager.cc:1415] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51686: name: "client-testtb" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "non_null_with_default" type: INT32 is_key: false is_nullable: false read_default_value: "90\000\000" encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { rows: "\004\001\000\t\000\000\000" } partition_schema { range_schema { columns { name: "key" } } } I0428 12:09:10.835557 10228 tablet_service.cc:1017] Processing CreateTablet for tablet 8a0809431eb840b9a71a9cb3591d2541 (table=client-testtb [id=1eb1f4c8454d45c485f7b779ea1ff324]), partition=RANGE (key) PARTITION VALUES < 9 I0428 12:09:10.835717 10227 tablet_service.cc:1017] Processing CreateTablet for tablet e3707c7829e840c7865e5f78c1585744 (table=client-testtb [id=1eb1f4c8454d45c485f7b779ea1ff324]), partition=RANGE (key) PARTITION VALUES >= 9 I0428 12:09:10.836239 10228 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet 8a0809431eb840b9a71a9cb3591d2541. 1 dirs total, 0 dirs full, 0 dirs failed I0428 12:09:10.836614 10227 data_dirs.cc:938] Could only allocate 1 dirs of requested 3 for tablet e3707c7829e840c7865e5f78c1585744. 1 dirs total, 0 dirs full, 0 dirs failed I0428 12:09:10.844729 10273 tablet_bootstrap.cc:439] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap starting. I0428 12:09:10.846640 10273 tablet_bootstrap.cc:590] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: No blocks or log segments found. Creating new log. I0428 12:09:10.849748 10273 tablet_bootstrap.cc:439] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: No bootstrap required, opened a new log I0428 12:09:10.849994 10273 ts_tablet_manager.cc:1053] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s I0428 12:09:10.850921 10273 raft_consensus.cc:340] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.851246 10273 raft_consensus.cc:366] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0428 12:09:10.851367 10273 raft_consensus.cc:705] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Initialized, Role: FOLLOWER I0428 12:09:10.851670 10273 consensus_queue.cc:229] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.852020 10273 raft_consensus.cc:380] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0428 12:09:10.852155 10273 raft_consensus.cc:472] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0428 12:09:10.852298 10273 raft_consensus.cc:2886] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Advancing to term 1 I0428 12:09:10.855520 10273 raft_consensus.cc:494] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.855862 10273 leader_election.cc:310] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b7784188f2497aabfce250a5c87ec4; no voters: I0428 12:09:10.856923 10273 leader_election.cc:296] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 1 election: Requested vote from peers I0428 12:09:10.857142 10275 raft_consensus.cc:2640] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Leader election won for term 1 I0428 12:09:10.858513 10273 ts_tablet_manager.cc:1081] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s I0428 12:09:10.858942 10273 tablet_bootstrap.cc:439] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap starting. I0428 12:09:10.860997 10273 tablet_bootstrap.cc:590] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: No blocks or log segments found. Creating new log. I0428 12:09:10.862071 10275 raft_consensus.cc:667] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Running, Role: LEADER I0428 12:09:10.862620 10275 consensus_queue.cc:206] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.864363 10273 tablet_bootstrap.cc:439] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: No bootstrap required, opened a new log I0428 12:09:10.864686 10273 ts_tablet_manager.cc:1053] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0428 12:09:10.867347 10273 raft_consensus.cc:340] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.867767 10273 raft_consensus.cc:366] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0428 12:09:10.867943 10273 raft_consensus.cc:705] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Initialized, Role: FOLLOWER I0428 12:09:10.867992 10153 catalog_manager.cc:3987] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 reported cstate change: term changed from 0 to 1, leader changed from to 44b7784188f2497aabfce250a5c87ec4 (127.8.60.1). New cstate: current_term: 1 leader_uuid: "44b7784188f2497aabfce250a5c87ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } health_report { overall_health: HEALTHY } } } I0428 12:09:10.868376 10273 consensus_queue.cc:229] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.868873 10273 raft_consensus.cc:380] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0428 12:09:10.869063 10273 raft_consensus.cc:472] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0428 12:09:10.869266 10273 raft_consensus.cc:2886] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 0 FOLLOWER]: Advancing to term 1 I0428 12:09:10.873401 10273 raft_consensus.cc:494] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.873890 10273 leader_election.cc:310] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b7784188f2497aabfce250a5c87ec4; no voters: I0428 12:09:10.874310 10273 leader_election.cc:296] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 1 election: Requested vote from peers I0428 12:09:10.874413 10275 raft_consensus.cc:2640] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Leader election won for term 1 I0428 12:09:10.874753 10275 raft_consensus.cc:667] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Running, Role: LEADER I0428 12:09:10.875196 10273 ts_tablet_manager.cc:1081] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s I0428 12:09:10.875284 10275 consensus_queue.cc:206] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:10.878392 10153 catalog_manager.cc:3987] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 reported cstate change: term changed from 0 to 1, leader changed from to 44b7784188f2497aabfce250a5c87ec4 (127.8.60.1). New cstate: current_term: 1 leader_uuid: "44b7784188f2497aabfce250a5c87ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } health_report { overall_health: HEALTHY } } } W0428 12:09:12.517338 10286 log.cc:669] Time spent T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Append to log took a long time: real 0.079s user 0.067s sys 0.008s I0428 12:09:13.242863 10288 rpcz_store.cc:269] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55498 (ReqId={client: 8917f2fd3ede4ef3b26aa9e3b68f2a1b, seq_no=1, attempt_no=0}) took 1215ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":223,"num_ops":9877,"prepare.queue_time_us":76,"prepare.run_cpu_time_us":387871,"prepare.run_wall_time_us":399404,"replication_time_us":85538,"spinlock_wait_cycles":1920}]]} I0428 12:09:13.270576 8432 tablet_server.cc:165] TabletServer shutting down... I0428 12:09:13.296628 8432 ts_tablet_manager.cc:1126] Shutting down tablet manager... I0428 12:09:13.297000 8432 tablet_replica.cc:264] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: stopping tablet replica I0428 12:09:13.297379 8432 raft_consensus.cc:2143] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 LEADER]: Raft consensus shutting down. I0428 12:09:13.297868 8432 raft_consensus.cc:2170] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Raft consensus is shut down! I0428 12:09:13.329869 8432 tablet_replica.cc:264] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: stopping tablet replica I0428 12:09:13.330294 8432 raft_consensus.cc:2143] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 LEADER]: Raft consensus shutting down. I0428 12:09:13.330646 8432 raft_consensus.cc:2170] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Raft consensus is shut down! I0428 12:09:13.342486 8432 tablet_server.cc:179] TabletServer shutdown complete. I0428 12:09:13.342618 8432 tablet_server.cc:165] TabletServer shutting down... I0428 12:09:13.342921 8432 tablet_server.cc:179] TabletServer shutdown complete. I0428 12:09:13.362828 8432 system_ntp.cc:202] NTP initialized. Skew: 500ppm Current error: 106249us I0428 12:09:13.363196 8432 fs_manager.cc:263] Metadata directory not provided I0428 12:09:13.363323 8432 fs_manager.cc:266] Using existing metadata directory in first data directory I0428 12:09:13.366196 8432 fs_manager.cc:399] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s I0428 12:09:13.367367 8432 file_cache.cc:466] Constructed file cache lbm with capacity 419430 W0428 12:09:13.368721 10267 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.8.60.1:38987, user_credentials={real_user=slave}} blocked reactor thread for 124751us I0428 12:09:13.373231 8432 fs_manager.cc:419] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0428 12:09:13.373361 8432 fs_manager.cc:436] Opened local filesystem: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root uuid: "44b7784188f2497aabfce250a5c87ec4" format_stamp: "Formatted at 2019-04-28 12:09:10 on dist-test-slave-dist-test-slave-lsq8" I0428 12:09:13.373661 8432 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root metadata directory: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root 1 data directories: /tmp/dist-test-task0fzxtl/test-tmp/client-test.4.ClientTest.TestServerTooBusyRetry.1556453332237234-8432/minicluster-data/ts-0-root/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0428 12:09:13.397341 8432 ts_tablet_manager.cc:344] Loading tablet metadata (0/2 complete) I0428 12:09:13.404422 8432 ts_tablet_manager.cc:356] Loaded tablet metadata (2 total tablets, 2 live tablets) I0428 12:09:13.404588 8432 ts_tablet_manager.cc:362] Registering tablets (0/2 complete) I0428 12:09:13.408200 10301 tablet_bootstrap.cc:439] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap starting. I0428 12:09:13.410636 8432 ts_tablet_manager.cc:376] Registered 2 tablets I0428 12:09:13.419356 10301 tablet_bootstrap.cc:439] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=9 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0428 12:09:13.419888 10301 tablet_bootstrap.cc:439] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap complete. I0428 12:09:13.420202 10301 ts_tablet_manager.cc:1053] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s I0428 12:09:13.421242 10301 raft_consensus.cc:340] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:13.421715 10301 raft_consensus.cc:705] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Initialized, Role: FOLLOWER I0428 12:09:13.422124 10301 consensus_queue.cc:229] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:13.422526 10301 raft_consensus.cc:380] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0428 12:09:13.422685 10301 raft_consensus.cc:472] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0428 12:09:13.422876 10301 raft_consensus.cc:2886] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Advancing to term 2 I0428 12:09:13.426766 10301 raft_consensus.cc:494] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:13.427171 10301 leader_election.cc:310] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b7784188f2497aabfce250a5c87ec4; no voters: I0428 12:09:13.428216 10309 raft_consensus.cc:2640] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 2 FOLLOWER]: Leader election won for term 2 I0428 12:09:13.428766 10301 leader_election.cc:296] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 2 election: Requested vote from peers I0428 12:09:13.430105 10309 raft_consensus.cc:667] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [term 2 LEADER]: Becoming Leader. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Running, Role: LEADER I0428 12:09:13.430420 10309 consensus_queue.cc:206] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:13.431728 10301 ts_tablet_manager.cc:1081] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s I0428 12:09:13.432168 10301 tablet_bootstrap.cc:439] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap starting. I0428 12:09:13.469449 8432 rpc_server.cc:205] RPC server started. Bound to: 127.8.60.1:38987 I0428 12:09:13.471639 8432 webserver.cc:308] Webserver started at http://127.8.60.1:44795/ using document root and password file I0428 12:09:13.489202 10363 heartbeater.cc:345] Connected to a master server at 127.8.60.62:40873 I0428 12:09:13.489429 10363 heartbeater.cc:426] Registering TS with master... I0428 12:09:13.489830 10363 heartbeater.cc:475] Master 127.8.60.62:40873 requested a full tablet report, sending... I0428 12:09:13.491854 10153 ts_manager.cc:159] Re-registered known tserver with Master: 44b7784188f2497aabfce250a5c87ec4 (127.8.60.1:38987) I0428 12:09:13.492802 10153 catalog_manager.cc:3987] T 8a0809431eb840b9a71a9cb3591d2541 P 44b7784188f2497aabfce250a5c87ec4 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "44b7784188f2497aabfce250a5c87ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } health_report { overall_health: HEALTHY } } } I0428 12:09:13.504786 10153 master_service.cc:273] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51692 I0428 12:09:13.517423 10363 heartbeater.cc:467] Master 127.8.60.62:40873 was elected leader, sending a full tablet report... W0428 12:09:13.590703 10301 log.cc:669] Time spent T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Append to log took a long time: real 0.075s user 0.076s sys 0.000s I0428 12:09:14.980334 10301 tablet_bootstrap.cc:439] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=9991 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0428 12:09:14.981637 10301 tablet_bootstrap.cc:439] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Bootstrap complete. I0428 12:09:14.982091 10301 ts_tablet_manager.cc:1053] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Time spent bootstrapping tablet: real 1.550s user 1.473s sys 0.071s I0428 12:09:14.983062 10301 raft_consensus.cc:340] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:14.983364 10301 raft_consensus.cc:705] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Initialized, Role: FOLLOWER I0428 12:09:14.983701 10301 consensus_queue.cc:229] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:14.984038 10301 raft_consensus.cc:380] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0428 12:09:14.984184 10301 raft_consensus.cc:472] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0428 12:09:14.984345 10301 raft_consensus.cc:2886] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 1 FOLLOWER]: Advancing to term 2 I0428 12:09:14.987196 10301 raft_consensus.cc:494] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:14.987541 10301 leader_election.cc:310] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b7784188f2497aabfce250a5c87ec4; no voters: I0428 12:09:14.988296 10301 leader_election.cc:296] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [CANDIDATE]: Term 2 election: Requested vote from peers I0428 12:09:14.988507 10371 raft_consensus.cc:2640] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 2 FOLLOWER]: Leader election won for term 2 I0428 12:09:14.989573 10301 ts_tablet_manager.cc:1081] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s I0428 12:09:14.990018 10371 raft_consensus.cc:667] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [term 2 LEADER]: Becoming Leader. State: Replica: 44b7784188f2497aabfce250a5c87ec4, State: Running, Role: LEADER I0428 12:09:14.990443 10371 consensus_queue.cc:206] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } } I0428 12:09:14.999480 10153 catalog_manager.cc:3987] T e3707c7829e840c7865e5f78c1585744 P 44b7784188f2497aabfce250a5c87ec4 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "44b7784188f2497aabfce250a5c87ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7784188f2497aabfce250a5c87ec4" member_type: VOTER last_known_addr { host: "127.8.60.1" port: 38987 } health_report { overall_health: HEALTHY } } } W0428 12:09:34.785876 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800044840 after lost signal to thread 8435 W0428 12:09:34.786664 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000726c0 after lost signal to thread 10185 W0428 12:09:34.787297 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800095aa0 after lost signal to thread 10361 I0428 12:11:10.754742 10143 maintenance_manager.cc:306] P ee7889a3f0d645a8aa308d11ba84d9ce: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033380 I0428 12:11:10.802081 10142 maintenance_manager.cc:501] P ee7889a3f0d645a8aa308d11ba84d9ce: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.046s user 0.011s sys 0.030s Metrics: {"bytes_written":6968,"cfile_init":1,"data dirs.queue_time_us":1400,"data dirs.run_cpu_time_us":680,"data dirs.run_wall_time_us":1569,"drs_written":1,"lbm_read_time_us":154,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1286,"lbm_writes_lt_1ms":27,"rows_written":5,"thread_start_us":3324,"threads_started":2,"wal-append.queue_time_us":1111} I0428 12:11:10.804054 10143 maintenance_manager.cc:306] P ee7889a3f0d645a8aa308d11ba84d9ce: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 504 bytes on disk I0428 12:11:10.805398 10142 maintenance_manager.cc:501] P ee7889a3f0d645a8aa308d11ba84d9ce: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":170,"lbm_reads_lt_1ms":4} I0428 12:11:13.554610 10364 maintenance_manager.cc:306] P 44b7784188f2497aabfce250a5c87ec4: Scheduling FlushMRSOp(8a0809431eb840b9a71a9cb3591d2541): perf score=0.033367 I0428 12:11:13.589102 10300 maintenance_manager.cc:501] P 44b7784188f2497aabfce250a5c87ec4: FlushMRSOp(8a0809431eb840b9a71a9cb3591d2541) complete. Timing: real 0.034s user 0.015s sys 0.015s Metrics: {"bytes_written":4841,"cfile_init":1,"data dirs.queue_time_us":2249,"data dirs.run_cpu_time_us":390,"data dirs.run_wall_time_us":783,"drs_written":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1223,"lbm_writes_lt_1ms":27,"rows_written":9,"thread_start_us":3952,"threads_started":2,"wal-append.queue_time_us":2343} I0428 12:11:13.590760 10364 maintenance_manager.cc:306] P 44b7784188f2497aabfce250a5c87ec4: Scheduling UndoDeltaBlockGCOp(8a0809431eb840b9a71a9cb3591d2541): 206 bytes on disk I0428 12:11:13.591854 10300 maintenance_manager.cc:501] P 44b7784188f2497aabfce250a5c87ec4: UndoDeltaBlockGCOp(8a0809431eb840b9a71a9cb3591d2541) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":158,"lbm_reads_lt_1ms":4} W0428 12:11:14.161433 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b1240 after lost signal to thread 8435 W0428 12:11:14.162724 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b1200 after lost signal to thread 10185 W0428 12:11:14.164219 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001afb80 after lost signal to thread 10361 I0428 12:11:15.098455 10364 maintenance_manager.cc:306] P 44b7784188f2497aabfce250a5c87ec4: Scheduling FlushMRSOp(e3707c7829e840c7865e5f78c1585744): perf score=0.033364 I0428 12:11:17.209082 10300 maintenance_manager.cc:501] P 44b7784188f2497aabfce250a5c87ec4: FlushMRSOp(e3707c7829e840c7865e5f78c1585744) complete. Timing: real 2.110s user 2.017s sys 0.047s Metrics: {"bytes_written":154532,"cfile_init":1,"data dirs.queue_time_us":1186,"data dirs.run_cpu_time_us":258,"data dirs.run_wall_time_us":915,"drs_written":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3273,"lbm_writes_lt_1ms":36,"rows_written":9991,"spinlock_wait_cycles":21376,"thread_start_us":4833,"threads_started":2,"wal-append.queue_time_us":3294} I0428 12:11:17.211030 10364 maintenance_manager.cc:306] P 44b7784188f2497aabfce250a5c87ec4: Scheduling UndoDeltaBlockGCOp(e3707c7829e840c7865e5f78c1585744): 63094 bytes on disk I0428 12:11:17.212177 10300 maintenance_manager.cc:501] P 44b7784188f2497aabfce250a5c87ec4: UndoDeltaBlockGCOp(e3707c7829e840c7865e5f78c1585744) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4} W0428 12:11:21.527719 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000c06c0 after lost signal to thread 8435 W0428 12:11:21.528949 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080010c4e0 after lost signal to thread 10185 W0428 12:11:21.530421 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800080f40 after lost signal to thread 10361 W0428 12:11:52.430128 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08001020a0 after lost signal to thread 8435 W0428 12:11:52.431684 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000726a0 after lost signal to thread 10185 W0428 12:11:52.433346 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080014c600 after lost signal to thread 10361 W0428 12:12:52.487504 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000c4800 after lost signal to thread 8435 W0428 12:12:52.489984 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001aa820 after lost signal to thread 10185 W0428 12:12:52.492389 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000bcde0 after lost signal to thread 10361 W0428 12:13:08.720708 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08001b2260 after lost signal to thread 8435 W0428 12:13:08.723304 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000d3d20 after lost signal to thread 10185 W0428 12:13:08.725453 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000f9880 after lost signal to thread 10361 W0428 12:13:33.580255 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080019a9e0 after lost signal to thread 8435 W0428 12:13:33.582811 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001a63a0 after lost signal to thread 10185 W0428 12:13:33.585412 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b12a0 after lost signal to thread 10361 W0428 12:13:41.181787 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000c2060 after lost signal to thread 8435 W0428 12:13:41.183740 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001aa8e0 after lost signal to thread 10185 W0428 12:13:41.185993 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001aa240 after lost signal to thread 10361 W0428 12:14:07.135526 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000348a0 after lost signal to thread 8435 W0428 12:14:07.139647 10297 debug-util.cc:397] Leaking SignalData structure 0x7b0800030f40 after lost signal to thread 10185 W0428 12:14:07.142489 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b05e0 after lost signal to thread 10361 W0428 12:14:12.323207 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001c0f80 after lost signal to thread 8435 W0428 12:14:12.325389 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001aa840 after lost signal to thread 10185 W0428 12:14:12.327837 10297 debug-util.cc:397] Leaking SignalData structure 0x7b0800108e20 after lost signal to thread 10361 W0428 12:14:29.192219 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080014c8e0 after lost signal to thread 8435 W0428 12:14:29.194649 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800087a20 after lost signal to thread 10185 W0428 12:14:29.197175 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000a36c0 after lost signal to thread 10361 W0428 12:14:33.673511 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000ab3e0 after lost signal to thread 8435 W0428 12:14:33.676553 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000ae540 after lost signal to thread 10185 W0428 12:14:33.679672 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800091f40 after lost signal to thread 10361 W0428 12:14:54.168947 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080018f340 after lost signal to thread 8435 W0428 12:14:54.172576 10297 debug-util.cc:397] Leaking SignalData structure 0x7b0800196420 after lost signal to thread 10185 W0428 12:14:54.176089 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b0dc0 after lost signal to thread 10361 W0428 12:15:22.705790 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800094140 after lost signal to thread 8435 W0428 12:15:22.708673 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000a04c0 after lost signal to thread 10185 W0428 12:15:22.711470 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000aa980 after lost signal to thread 10361 W0428 12:16:53.905236 10297 debug-util.cc:397] Leaking SignalData structure 0x7b0800107b20 after lost signal to thread 8435 W0428 12:16:53.909344 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b11e0 after lost signal to thread 10185 W0428 12:16:53.913748 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001a9ee0 after lost signal to thread 10361 W0428 12:17:05.582602 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000728e0 after lost signal to thread 8435 W0428 12:17:05.587178 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000ae560 after lost signal to thread 10185 W0428 12:17:05.591588 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000d5200 after lost signal to thread 10361 W0428 12:17:17.014639 10297 debug-util.cc:397] Leaking SignalData structure 0x7b0800033f80 after lost signal to thread 8435 W0428 12:17:17.018281 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080018f000 after lost signal to thread 10185 W0428 12:17:17.025362 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000b92c0 after lost signal to thread 10361 W0428 12:18:02.659081 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080004aa40 after lost signal to thread 8435 W0428 12:18:02.662873 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800087ea0 after lost signal to thread 10185 W0428 12:18:02.666901 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000d3d40 after lost signal to thread 10361 W0428 12:18:40.667807 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800197620 after lost signal to thread 8435 W0428 12:18:40.672436 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000db380 after lost signal to thread 10185 W0428 12:18:40.678407 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800088fa0 after lost signal to thread 10361 W0428 12:18:42.848657 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001aabe0 after lost signal to thread 8435 W0428 12:18:42.853307 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080018f1c0 after lost signal to thread 10185 W0428 12:18:42.859366 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080019f680 after lost signal to thread 10361 W0428 12:19:07.507511 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080014c3c0 after lost signal to thread 8435 W0428 12:19:07.512302 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800095300 after lost signal to thread 10185 W0428 12:19:07.517994 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080014b340 after lost signal to thread 10361 W0428 12:20:07.406752 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000c2ba0 after lost signal to thread 8435 W0428 12:20:07.412693 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080019f2a0 after lost signal to thread 10185 W0428 12:20:07.418191 10297 debug-util.cc:397] Leaking SignalData structure 0x7b080018f060 after lost signal to thread 10361 W0428 12:20:23.873589 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000fc360 after lost signal to thread 8435 W0428 12:20:23.878401 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08000ccf20 after lost signal to thread 10185 W0428 12:20:23.884522 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800051ae0 after lost signal to thread 10361 W0428 12:22:03.715726 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000f9280 after lost signal to thread 8435 W0428 12:22:03.721261 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08001b0e40 after lost signal to thread 10185 W0428 12:22:03.727725 10297 debug-util.cc:397] Leaking SignalData structure 0x7b08000b7460 after lost signal to thread 10361 W0428 12:22:11.928373 10139 debug-util.cc:397] Leaking SignalData structure 0x7b0800044be0 after lost signal to thread 8435 W0428 12:22:11.933187 10139 debug-util.cc:397] Leaking SignalData structure 0x7b080018f3c0 after lost signal to thread 10185 W0428 12:22:11.939275 10139 debug-util.cc:397] Leaking SignalData structure 0x7b08001b3480 after lost signal to thread 10361 ==8432==ThreadSanitizer: Thread limit (8128 threads) exceeded. Dying.